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16  Ab,"°c'  ' This  report  covers  personnel  research  efforts  during  the  past  nine  years  ' 
directed  toward  improving  the  selection  of  applicants  to  work  in  the  Air  Traffic  Cor.- 
|trol  occupation.  The  report  summarizes  the  various  research  efforts  and  makes 
[specific  recommendations  for  changes  in  present  0PM  tests  through  which  applicants 
qualify  for  employment  consideration  in  the  ATC  occupation,  and  the  rating  and  ranking 
procedures  used  to  establish  a  register  of  eligible  applicants  for  appointment  to  the 
occupation.  I 

New  tests  developed  for  inclusion  in  a  revised  ATC  test  battery  include  the  Multiplex  | 
Controller  Aptitude  Test  (MCAT)  and  an  Occupational  Knowledge  Test  (0KT) .  Both  tests  j 
are  paper-pencil,  machine  scorable,  multiple  choice  type  tests.  The  MCAT  presents 
simulations  of  air  traffic  flow  in  each  item.  The  applicant  must  solve  air  traffic 

problems  or  a  related  cognitive  skill  problem  in  completing  each  item.  The  OKI  ; 

measures  important  aspects  of  knowledge  associated  with  prior  experience  that  are 
related  to  successful  performance  as  an  air  traffic  control  specialist. 

The  revised  test  battery  recommended  to  0FM  includes  the  Multiplex  Controller  Aptitude 
Test  and  the  present  0PM  test,  Abstract  Reasoning  and  Letter  Sequence.  The  Occupa-  j 

tional  Knowledge  Test  is  used  for  granting  additional  earned  credit  in  place  of  the 

present  0PM  Rating  Guide.  This  new  test  battery  meets  all  professional  requirements 
for  an  effective  test  including  fairness.*: _ 
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SELECTION  OF  APPLICANTS  FOR  THE  AIR  TRAFFIC  CONTROL  OCCUPATION 


INTRODUCTION 


This  report  has  been  prepared  to  consolidate  various  personnel  research 
efforts  during  the  past  9  years  directed  toward  improving  the  selection  of 
applicants  to  work  in  the  air  traffic  control  (ATC)  occupation.  The  position 
classification  standards  for  the  Air  Traffic  Control  Specialist  (ATCS)  Series 
(GS-2152)  established  by  the  Office  of  Personnel  Management  (OPM)  define  three 
najor  options  within  the  occupation. 

.  Flight  Service  Stations  (FSS)  which  provide  preflight  briefing  and 
assistance  and  advisory  services  to  pilots  during  flight; 

.  Terminals  which  provide  control  and  separation  of  air  traffic  at 
airports,  and, 

.  Centers  which  provide  control  and  separation  of  enroute  air  traffic. 

The  Federal  Aviation  Administration  (FAA)  is  the  major  employing  agency  for 
ATCSs  with  a  workforce  of  about  26,000  enployees  in  the  GS-2152  series.  They 
represent  almost  50  percent  of  the  total  FAA  full-time  staffing.  Selection 
and  hiring  of  applicants  for  the  occupation  by  FAA  over  the  past  5  years  has 
ranged  between  1,700  and  2,000  new  enployees  annually  and  can  be  expected  to 
continue  at  about  the  same  levels.  The  Department  of  Defense  (DOD)  employs  a 
total  of  about  300  civilian  air  traffic  personnel.  DOD  nay  select  applicants 
from  the  same  OPM  register  used  by  the  FAA,  although  their  use  of  the  register 
is  minimal.  Positions  as  ATCSs  are  essentially  nonexistent  in  the  private 
sector. 

This  report  summarizes  the  various  research  efforts  and,  based  on  the  results, 
makes  specific  recommendations  for  changes  in; 

.  Present  OPM  tests  through  which  applicants  qualify  for  enployment  con¬ 
sideration  in  the  ATC  occupations,  and, 

.  The  rating  and  ranking  procedures  used  to  establish  a  register  of 
eligible  applicants  for  appointment  to  the  occupation. 

The  research  documents  and  studies  referenced  in  the  text  of  this  report  are 
identified  in  the  bibliography. 
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HISTORICAL  DEVELOPMENT  OF  PRESENT  SELECTION  PROCEDURES 


Selection  of  ATC  personnel  has  been  an  area  of  concern  for  a  number  of 
agencies  over  a  considerable  time  period.  Continuing  research  since  the  early 
1950's  has  been  directed  toward  inproving  the  processes  by  which  applicants  or 
candidates  for  civil  and  military  ATCSs  are  evaluated,  tested,  and  selected. 
Reference  (1)  provides  a  survey  of  selected  research  studies  on  ATC  selection 
during  the  period  1952-1972. 

In  August  1960,  the  FAA  Civil  Aeromedical  Institute  (CAMI)  began  administering 
a  heterogeneous  battery  of  commercially  available  tests  on  an  experimental 
basis  to  newly  selected  ATCSs  at  the  FAA  Academy  in  Oklahoma  City.  After  the 
9  week  Academy  ATC  training  course,  the  student's  average  academic  test  scores 
and  average  laboratory  scores  were  summed  to  form  a  composite  which  was  corre¬ 
lated  (Pearson  product  -  moment  formula)  with  the  composite  of  the  experimen¬ 
tal  aptitude  test  scores.  The  coefficients  ranged  fron.  .35  to  .54.  Based  on 
these  findings,  it  was  concluded  that  aptitude  tests  could  improve  selection 
of  ATC  applicants  who  would  be  successful  in  the  occupation. 

The  commercial  tests  used  experimentally  were  considered  to  be  more  suscep¬ 
tible  to  compromise  than  those  controlled  by  the  Civil  Service  Commission 
(CSC).  Consequently,  those  commercial  tests  that  showed  the  most  promise  were 
used  to  identify  CSC  tests  which  were  similar  in  factor  content.  Table  I 
identifies  the  five  tests  and  the  zero-order  correlations  derived  from,  these 
studies  (2).  These  CSC  tests  and  an  additional  Air  Traffic  Problems  Test 
developed  for  FAA  were  then  employed,  beginning  in  1961,  in  another  series  of 


testing 

sessions  at  the  FAA  Academy,  i 

Subsequent  regression  analysis  resulted 

in  identifying  five  best  predictors  of 

training  success 

using  course  grade  and 

pass  or 

fail  status  of  ATC  students. 

TABLE  1  CSC 

TEST  VALIDITIES 

Testing 

Dates: 

(N  =  143  -  pass) 

September  1960-June  1962 

(N  =  183) 

(N  =  40  -  fail) 

CSC  Tests1 

Course  Grade 

Pass/Fail 

CSC  51 

Spatial  Patterns 

.37** 

.27** 

CSC  24 

Computations 

.28** 

.16* 

CSC  157 

Abstract  Reasoning 

.28** 

.18* 

CSC  157 

Letter  Sequence 

.55** 

.45** 

CSC  135 

Oral  Directions 

.23** 

.23** 

CSC  540 

Air  Traffic  Problems 

.41** 

.29** 

*p  .05;  **p  *1  .01 
^Test  number  assigned  by  CSC 
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Beginning  in  1962,  the  test  battery  identified  in  Table  I  served  as  a  major 
determinant  in  the  selection  of  one-third  of  the  applicants  with  little  or  no 
prior  ATC  related  experience.  Applicants  with  experience  continued  to  be 
selected  without  a  requirement  for  formal  testing.  CAMI  continued  to  collect 
and  analyze  data  on  the  test  battery  after  its  adoption  by  the  CSC.  In 
January  1964,  the  CSC  battery  was  introduced  as  a  means  to  qualify  all  appli¬ 
cants  regardless  of  their  previous  experience,  although  air  traffic  related 
experience  continued  to  be  used  as  a  ranking  factor  for  placing  applicants  on 
the  CSC  register  for  selection. 

In  October  1968,  further  modifications  were  made  to  the  ATC  selection  system 
aimed  at  relieving  a  shortage  of  ATC  personnel  in  a  rapidly  expanding  ATC 
system.  A  major  change  provided  for  hiring  applicants  with  specialized  ATC 
experience  (particularly  in  military  radar  control)  at  a  higher  grade  (GS-9/ 
GS-11)  without  requiring  that  they  take  the  CSC  test  battery. 

These  ATCS  selection  procedures  renained  essentially  unchanged  from  1968  until 
1973  when  a  maximum  entry  age  (31  years)  for  the  Center  and  Terminal  options 
was  established  by  Public  Law  92-297.  At  this  time,  CSC  and  FAA  re-estab¬ 
lished  the  requirement  that  all  ATC  applicants  take  the  CSC  test  battery  and 
achieve  a  score  of  210  or  higher  for  employment  eligibility.  From  1973  to  the 
present,  the  ATC  selection  procedures  have  remained  essentially  the  same. 

The  methods  and  standards  for  establishing  ATC  employment  eligibility  have 
varied  from  time-to-time  since  1962.  However,  the  total  selection  process  has 
typically  involved  two  elements:  the  use  of  the  CSC  test  battery  to  determine 
eligibility  for  employment;  and  the  weighting  and  use  of  prior  aviation- 
related  experience  either  directly  or  indirectly  in  selecting  from  among  those 
applicants  eligible  for  employment.  References  (2),  (3),  (4),  and  (5)  provide 
sources  for  more  detailed  information  on  the  development  of  ATC  selection  pro¬ 
cedures  and  (Chapter  I  of  reference  6)  presents  a  more  extensive  discussion  of 
current  selection  processes. 
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PROBLEM  AND  RESEARCH  OBJECTIVES 


Attrition  of  ATCSs  during  the  developmental  training  period  has  been  an  area 
of  continuing  concern  for  the  FAA.  Analysis  of  training  losses  over  the  past 
9  years  highlights  the  fact  that  attrition  ranged  between  25  and  40  percent 
during  the  2  to  5  year  training  period  required  to  reach  the  full  performance 
level  (FPL).  The  major  part  of  this  attrition  resulted  from  the  inability  of 
trainees  to  acquire  and  demonstrate  the  skills  and  knowledge  to  progress 
satisfactorily  through  the  training  program.  This  has  been  the  case  indepen¬ 
dent  of  whether  or  not  the  training  has  been  conducted  on  a  completely  decen¬ 
tralized  basis  or  when  selected  phases  of  training  have  been  conducted  at  the 
FAA  Academy  on  a  centralized  basis. 

Attrition  rates  of  this  magnitude  involved  investment  losses  of  millions  of 
dollars  annually  to  the  FAA  in  salaries,  training,  and  travel  costs.  Of  equal 
importance  to  maintaining  an  adequate  staff  of  full  performance  controllers  is 
the  irretrievable  loss  of  "time,"  since  a  new  applicant  hired  as  a  replacement 
must  start  at  the  beginning  of  the  training  cycle. 

This  significant  investment  loss  to  the  FAA  has  two  major  components.  First, 
the  screening  and  selection  criteria  used  to  determine  which  applicants  to 
hire  for  the  occupation;  second,  the  length  of  time  taken  to  identify  trainees 
who  do  not  demonstrate  the  ability  to  progress  to  the  full  performance  level 
required  by  the  occupation. 

To  establish  more  comprehensive  and  uniform  training,  and  to  provide  earler 
identification  of  trainees  who  do  not  demonstrate  the  necessary  skills  and 
abilities,  the  FAA  re-established  a  centralized  ATC  Initial  Qualification 
Training  Program  at  the  FAA  Academy  for  the  terminal  and  center  options  and 
restructured  the  follow-on  training  conducted  at  the  ATC  facilities  in  January 
1976.  Selected  phases  of  training  leading  to  FPL  are  conducted  on  a  pass/fail 
basis  using  defined  job-related  training  outcomes  and  standards.  A  comparable 
pass/fail  training  program  was  established  for  the  FSS  option  in  September 
1978. 

Terminal,  center,  and  FSS  initial  oualif ication  training  at  the  FAA  Academy 
consists  of  approximately  15  weeks  of  instruction  in  both  academic  and  labora¬ 
tory  environments  where  students  must  demonstrate  the  ability  to  learn  and 
apply  the  knowledge  and  skills  taught  at  the  Acadeny.  Failure  to  pass  the 
training  is  the  basis  for  removal  from  the  ATC  occupation  or  the  FAA.  While 
the  attrition  rate  fluctuates  from  class  to  class,  the  average  loss  rate 
during  the  initial  qualification  training  over  the  past  3  years  in  terminal 
and  center  training  has  been  24  percent.  Four  percent  of  the  students  volun¬ 
tarily  withdrew  from  training  and  20  percent  failed  the  course.  This  rate 
varies  for  each  year.  For  1977,  the  failure  rate  was  21  percent  and  with¬ 
drawal  5  percent;  for  1978,  failure  30  percent  and  withdrawal  8  percent;  and 
in  1979,  failures  were  28  percent  and  withdrawals  7  percent. 
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The  FAA  Academy  training  has  been  effective  in  providing  early  identification 
of  those  trainees  who  do  not  demonstrate  the  necessary  abilities  to  progress 
to  the  jorneyman  ATCS  level.  Between  1970  and  1976,  when  all  ATC  training  was 
conducted  on  a  decentralized  basis  at  the  individual  ATC  facilities,  the 
average  length  of  service  for  ATC  trainees  who  separated  from  the  occupation 
was  between  18  and  24  months.  With  initial  qualification  training  conducted 
at  the  FAA  Academy,  most  attrition  which  can  be  expected  to  occur  due  to  per¬ 
formance  takes  place  during  the  first  5  months  of  employment.  By  itself,  this 
change  in  FAA  ATC  training  provides  a  cost  avoidance  on  the  order  of  $10.0  to 
$12.0  million  a  year. 

There  still  remains,  however,  the  concern  with  the  overall  attrition  rate  of 
ATC  trainees.  For  exanple,  salary  and  training  costs  for  the  initial  ATC 
qualification  training  program  represent  about  $10,000  for  each  employee 
hired.  With  a  hiring  rate  of  1,800  new  employees  annually,  a  24  percent 
attrition  rate  equates  to  an  investment  loss  of  about  $43  million  each  year. 

Consequently,  the  primary  purpose  of  this  personnel  research  is  to  develop 
improved  selection  procedures  in  order  to  hire  those  applicants  who  clearly 
demonstrate  aptitute  for  ATC  and  whose  potential  for  success  is  greatest. 

References  (8) ,  (9) ,  and  (10)  provide  sources  for  more  detailed  information  on 
ATC  traininee  attrition,  costs,  and  training  programs. 
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ATC  JOB  ANALYSIS 


The  Dictionary  of  Occupational  Titles  provides  a  general  description  of  the 
functions  and  activities  of  ATCSs  in  each  of  the  options.  In  addition,  more 
detailed  job/task  analyses  have  been  conpleted  for  a  variety  of  purposes. 

.  Position  Classification.  In  1978,  the  CSC  conpleted  a  revised 

Classification  Standard  for  the  GS-2152  ATC  occupation  which  identifies 
knowledge,  skills,  and  aptitudes  required  in  the  FSS,  terminal  and 
center  options. 

.  Performance  Evaluation.  Between  1971  and  1975,  the  System  Development 
Corporation  under  contract  with  the  FAA  conpleted  a  detailed  job/task 
analysis  of  the  terminal  and  center  options,  to  identify  the  basic 
skills  needed  for  successful  performance  of  each  task  required  of  a 
controller,  as  a  basis  for  developing  objective  performance  standards 
and  measures.  The  results  are  currently  used  by  the  FAA  in  conducting 
"over  the  shoulder"  evaluations  and  extended  training  reviews  for  ATCSs 
under  the  ATC  Technical  Appraisal  Program  (TAP) . 

.  ATC  Work  Attributes.  As  part  of  a  broader  research  study  on  ATC  selec¬ 
tion,  Education  and  Public  Affairs  (EPA) ,  Inc.,  under  contract  with  the 
FAA,  conpleted  an  analysis  of  the  work  attributes  associated  with  the 
ATC  occupation  and  proposed  methods  for  their  measurement.  Their 
report  (7)  identifies  nine  major  demands  of  ATC  work  (e.g.,  cognitive 
requirements) ;  over  30  attributes  (e.g. ,  judgment  and  reasoning 
ability)  associated  with  job  demands?  and,  describes  one  or  more  ATC 
tasks  related  to  these  attributes. 

In  addition  to  these  more  recent  ATC  job  analyses,  others  going  back' to  the 
1950's  have  been  conducted  (1). 


OVERVIEW  OF  RESEARCH  STUDIES 


Personnel  research  related  to  different  facets  of  the  ATC  occupation  has  been 
a  continuing  FAA  program  since  the  early  1960's.  This  report,  however,  will 
focus  on  the  results  of  studies  directed  to  initial  selection  of  applicants, 
particularly  the  development  of  screening  tests  and  the  use  of  prior  aviation 
experience  in  the  selection  and  placement  procefs.  Over  the  past  9  years, 
studies  have  encompassed  samples  of  four  major  population  groups. 

.  FPL  ATCSs; 

.  Developmental  ATCSs  with  several  years  of  ATC  training  who  have  not 
reached  FPL  status; 

.  New  ATC  appointees  who  were  just  entering  the  occupation;  and, 

.  Applicants  for  ATC  work. 

While  a  number  of  interrelated  studies  have  been  conpleted,  perhaps  the  best 
way  to  provide  an  overview  is  to  briefly  identify  them,  in  a  chronological 
secuence  although  they  were  obtain  conducted  as  parallel  efforts. 

.  1972  -  FAA  ATCSs.  A  najor  study  (12)  of  some  14  experimental  tests  as 
candidates  for  improving  selection  of  ATCS  applicants  was  completed  by 
EPA.  The  study  was  based  on  concurrent  analysis  of  about  800  ATCSs 
covering  all  three  options  and  included  both  journeyman  and  new 
appointees  in  the  ATC  occupation. 

.  1977  -  FAA  ATCS.  Based  in  part  on  longitudinal  analysis  of  the  infor¬ 
mation  obtained  in  the  1972  study,  EPA  conpleted  a  comprehensive 
analysis  of  11  experimental  predictors  involving  over  3,000  enployed 
and  separated  ATCSs.  This  study  (13)  "'Iso  enconpassed  all  three  ATC 
options. 

.  1976-1977  -  Applicants  for  ATC  Work.  Since  no  information  was  avail¬ 
able  on  the  characteristics  of  the  population  which  applies  for  ATC 
work,  CSC,  at  FAA's  request,  administered  three  of  the  experimental 
tests  or  predictor  forms  along  with  the  present  CSC  test  battery  to 
about  7,500  applicants  for  ATC  work  when  the  exam  was  opened  in  the  FAA 
Eastern  and  Southern  Regions.  The  information  obtained  provided  a 
variety  of  necessary  data  on  the  characteristics  of  individuals  who 
express  an  interest  in  the  ATC  occupation  by  taking  the  qualifying  CSC 
test.  It  also  provided  a  basis  for  correcting  for  restriction  in  range 
in  the  statistical  analysis  of  CSC  tests  and  experimental  predictors 
administered  to  those  individuals  hired  for  ATC  work. 

.  1976-1978  -  New  Appointees  to  the  ATC  Occupation.  With  the  establish¬ 
ment  of  the  ATC  initial  qualification  training  at  the  FAA  Acadeny  in 
1976,  the  Office  of  Aviation  Medicine  and  CAMI  initiated  a  continuing 
program  of  experimental  test  development,  administration,  and  analyses 
of  data  on  new  hires  in  the  center  and  terminal  options.  Experimental 
tests,  including  those  used  in  the  1977  study  by  EPA  and  administered 
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to  the  applicant  population  by  CSC,  were  given  to  new  AT C  hires  at 
the  FAA  Academy  prior  to  the  beginning  of  training.  In  addition,  the 
CSC  test  battery  scores  were  obtained  for  each  individual  whenever 
possible.  About  3,000  new  ATC  hires  were  included  in  this  study  with 
complete  data  on  all  variables  for  over  1,800  appointees. 

Analysis  of  the  data  for  this  sample  together  with  the  informa¬ 
tion  obtained  on  the  7,000  applicants  by  CSC  in  1976-77  provided  the 
basis  for  identifying  those  existing  CSC  or  experimental  tests  which, 
in  combination,  resulted  in  the  best  prediction  of  success  in  ATC 
training.  It  should  also  be  pointed  out  that  this  sample  group  (and 
subsequent  ATC  training  classes)  provide  the  basis  for  longitudinal 
analysis  of  tests  and  training  scores  with  on-the-job  training  and 
success  as  employees  progress  toward  FPL  status. 

1978-1979  -  New  Appointees  to  the  ATC  Occupation.  Based  on  analyses 
completed  through  1977,  the  FAA  Office  of  Aviation  Medicine  developed 
parallel  forms  for  those  experimental  tests  which  provided  the  best 
predictive  capability.  Beginning  in  June  1978,  two  parallel  versions 
of  the  tests  were  administered  to  each  employee  entering  ATC  training 
between  June  1978  and  January  1979.  About  1,100  trainees  comprise 
the  sample  for  this  study.  The  data  provided  the  basis  for  item 
analysis  of  the  parallel  test  forms  and  together  with  other  variables, 
their  correlation  with  training  success  or  failure. 

1978  -  Applicants  for  ATC  work.  During  September  -  October  1978,  the 
CSC  reopened  the  ATC  examination  for  applicants.  At  FAA's  request, 

CSC  administered  the  parallel  forms  of  the  experimental  tests  used  at 
the  FAA  Academy  along  with  the  standard  CSC  ATC  test  battery.  CSC 
also  obtained  information  on  sex  and  ethnic  groups,  for  these 
applicants,  which  previously  had  been  unobtainable.  This  applicant 
population  was  comprised  of  about  7,000,  including  about  1,000  who 
took  the  examination  on  a  "walk-in"  basis  as  part  of  the  affirmative 
action  recruitment  program.  This  information  provided  a  basis  not 
only  for  item  analysis  of  parallel  test  forms  relative  to  the  applicant 
population,  but  an  opportunity  to  evaluate  the  effect  of  changes 
in  tl  ■  qualifying  test  battery  on  selection  alternatives  and  criteria. 


FIGURE  I.  SEVERAL  RELATIONSHIPS  BETWEEN  ATC  SELECTION  RESEARCH  STUDJE 


1972  STUDY  -  SELECTION  OF  AIR  TRAFFIC  CONTROL  SPECIALISTS 


Objectives.  The  focus  of  this  research  was  on  selection  as  related  to  job 
performance.  It  was  based  on  concurrent  validity  analysis.  The  primary 
questions  addressed  were: 


.  To  what'  extent  is  it  possible  to  predict  the  quality  of  job 

performance  of  a  journeyman  ATCS  from  a  battery  of  tests  adminis¬ 
tered  at  the  time  of  job  application? 


.  To  what  extent  can  improvement  be  achieved  by  assigning  applicants 
consistently  to  one  of  four  selected  ATC  options  (FSS,  Center,  IFR 
or  VFR  terminals)  and  within  these,  into  high  or  low  density  (activity) 
facilities? 

.  To  what  extent  do  the  measures  selected  to  accomplish  the  above 
objectives  affect  black  and  white  applicants  with  an  equal  degree 
of  fairness? 


Sample  Description.  The  total  sample  was  comprised  of  approximately  800 
employees  who  were  either  journeymen  ATC  specialists  (FPL)  or  new  ATC 
appointees  including  an  oversample  of  black  ATC  specialists.  Sample- 
selection  provided  for  distribution  between  four  types  of  ATC  facilities 
(Centers,  IFR  and  VFR  Terminals,  and  Flight  Service  Stations).  Samples 
for  Centers  and  IFR  terminals  were  further  stratified  between  high  and 
low  activity  facilities.  Racial  distribution,  including  the  oversample, 
was  93  percent  white  and  7  percent  black. 

Using  15  major  cities  in  the  U.  S. ,  the  sample  was  randomly  selected 
(except  for  the  oversample)  from  ATC  facilities  within  a  100-mile  radius 
of  the  15  hub  cities.  Sampling  was  controlled  to  exclude  ATC  specialists 
over  36  years  of  age  and  to  insure  that  FPL  Specialists  had  no  less  than 
3  years  nor  more  than  10  years  of  ATC  experience  with  FAA.  Because  of  the 
small  number  of  women  in  the  ATC  workforce,  it  was  not  possible  to  stratify 
the  sample  selection  based  on  sex.  Participation  on  the  part  of  ATC 
specialists  was  voluntary.  All  subjects  were  administered  a  battery  of 
paper-pencil  tests  or  forms  including  the  present  CSC  test  battery. 

In  addition,  a  subsample  of  about  260  ATC  Specialists  who  took  the 
paper-pencil  battery  were  administered  a  series  of  "psychomotor"  tests  in 
a  separate  testing  session  at  FAA  facilities  at  Oklahoma  City.  Of  the 
journeymen,  34  percent  took  the  psychomotor  tests  together  with  32  percent 
of  the  new  appointees.  Selection  of  this  subsample  was  controlled  for  region, 
type  of  ATC  facilities  and  variance  on  the  confidential  supervisory  per¬ 
formance  evaluation  which  was  used  as  the  criterion  measure.  Racial 
distribution  of  the  "psychomotor"  subsample  was  83  percent  white  and  17 
percent  black. 
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Predictors.  In  selecting  predictors  for  this  study,  four  objectives  were 
considered : 

.  They  should  cover  as  broadly  as  possible,  the  range  of  job 
and  worker  attributes  identified  with  the  ATC  occupation. 

.  Experimental  tests  should  not  substantially  overlap  areas  already 
covered  by  the  existing  CSC  test  battery  which  was  to  be  adminis¬ 
tered  to  the  ATC  Specialists. 

.  Experimental  tests  should  be  selected  which,  based  on  prior 

research,  appeared  to  have  potential  validity  for  ATC  selections. 

.  The  two  selected  test  batteries  (paper-pencil  and  psychomotor) 
should  not  require  more  than  8  hours  each  to  administer. 

Within  these  objectives,  the  following  were  selected: 

•  Paper-Pencil  Battery 

-  Aptitude  Tests 

Current  CSC  test  battery  (5  parts) 

Minimum  Coins  test 

Dial  and  Table  Reading  Test 

-  Knowledge  and  Interest  tests 

Dailey  Technical  and  Scholastic  Test 
ATC  General  Information  Test 

-  "Personality”  Tests 

Concept-Adjective  Test 
Closure  Speed  Test 

-  Background  Information 
Biographical  Inventory 

.  Psychorootor  Battery 

Directional  Headings  Test 
Hidden  Patterns  Test 
Press  Test 

Controller  Decisions  Evaluation  (CODE)  Test 
Multiple  Task  Performance  Test 
Compressed  Speech  Test 
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The  Paper-Pencil  Battery  was  administered  to  the  total  sample  group. 
Test  administrators  were  FAA  employees,  trained  by  Education  and  Public 
Affairs.  The  Psychomotor  Battery  was  given  to  the  subsampls  at  FAA 
facilities  in  Oklahoma  City  where  equipment  required  for  administration  was 
available.  Detailed  discussion  on  the  source,  nature  of  the  test,  method 
of  scoring,  reliability  and  validity  for  the  selected  tests  is  provided 
elsewhere  ii.’i.  The  contractor  obtained  and  kept  all  test  results 
and  information  on  individual  ATC  Specialists  on  a  confidential  basis. 

No  individual  data  were  provided  to  FAA. 


Criterion  Measure.  After  examining  a  number  of  alternatives,  a  Confidential 
Supervisory  Evaluation  form  was  used  as  the  criterion  against  which  experi¬ 
mental  tests  were  validated.  These  evaluations  were  obtained  by  the  con¬ 
tractor  directly  from  the  ATC  Specialists'  supervisor.  They  were  not  reviewed 
FAA  nor  were  copies  provided  to  the  agency.  The  contractor  retained  all 
individual  evaluation  data  on  a  confidential  basis.  The  evaluation  covered 
the  broad  areas  of  performance  within  a  number  of  task  behaviors  for  each 
area. 


Performance  Area  Task  1 1 


Knowledge  3 
Perception  6 
Comprehension  6 
Memory  2 
Communication  8 
Judgement  4 
Traffic  Management  Techniques  4 
Performance  Work  Stress  4 
Interpersonnel  Skills  3 
Other  personnel  skills  5 


In  addition  an  "Overall  Performance"  category  consisting  of  four  task 
items  and  a  summary  evaluation  were  obtained.  The  summary  evaluation 
was  a  seven-point  rating  scale  with  "1"  being  the  highest  performance  and 
"7"  the  lowest. 

Descriptive  Information.  Based  on  biographic  responses  from  304  FPL  ATC 
Specialists,  the  following  background  and  educational  information  was 
obtained : 

.  98?  were  men;  2%  women 

.  96?  had  prior  military  service 

.  72?  claimed  prior  (military)  experience  as  a  controller 

.  2?  claimed  experience  as  a  pilot 

.  48?  took  the  CSC  test  battery  for  appointment;  10?  more  than 


once, before  passing 


Analytical  Methodology.  Data  collected  were  subjected  to  a  variety  of 
statistical  treatments.  Among  the  principal  methods  were: 

.  Multiple  regression  analysis  in  which  test  measures  were 
correlated  with  performance  evaluation  measures  to  determine 
how  well  they  predicted  job  performance. 

.  Analysis  of  variance  for  testing  differences  between  groups; 
minority  and  non-minority;  journeymen  and  new  appointees; 
and,  those  hired  with  previous  aviation-related  experience 
compared  to  those  hired  without  such  experience. 


.  Multiple  discriminant  function  analysis,  in  which  tests  were 
evaluated  in  terms  of  their  ability  to  maximize  placement 
within  option  and  activity  levels. 


Results.  In  summary,  this  study  concluded  that: 

1.  Capacity  of  the  tests  to  predict  job  performance  of  journeymen 
ATC  specialists  produced  mixed  results.  The  CSC  test  battery  was 
marginal  in  predicting  job  performance,  as  measured  by 
supervisory  evaluations.  This  result,  however,  was  due  pri¬ 
marily  to  the  restriction  in  range  of  the  CSC  test  predictors. 

This  restriction  resulted  not  only  from  the  selection  processes 
but  also  from  the  screening  which  took  place  during  the  develop¬ 
mental  training  period  as  the  controllers  included  in  the  sample 
progressed  to  the  journeyman  (FPL)  level.  This  study  did  not 
attempt  to  correct  for  the  range  restriction  and  consequently 
the  predictor  relationship  of  the  CSC  Tests  to  the  job  per¬ 
formance  criteria  was  not  determined.  The  psychomotor  tests 
developed  consistently  significant  correlations  with  supervisory 
job  performance  evaluations. 

2.  By  combining  paper-pencil  and  psychomotor  tests,  the  analysis 
indicated  that  it  was  possible  to  improve  assignments  to  the 
different  ATC  options  based  on  tests  results  and,  with  additional 
tests,  to  high  or  low  density  center  or  IFR  terminals. 

3.  If  relevance  to  the  job  is  the  primary  factor  in  determining 
test  acceptability,  the  tests  proposed  from  the  research  met 
that  criterion.  They  also  predicted  job  performance  equally 
well  for  each  racial  group  (black  and  white).  However,  blacks 

as  a  group  consistently  scored  lower  on  the  tests  than  did  whites. 
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This  study  provided  FAA  with  valuable  information  and  insights  on  the 
problems  associated  with  selection  and  placement  of  applicants  for  ATC 
work.  However,  action  on  the  research  results  and  recommendations  was 
deferred  due  to  a  number  of  considerations  including  the  complexity  of 
the  specialized  equipment  required  for  the  psychomotor  tests,  logistical 
difficulties  in  test  administration  and  complexity  in  test  scoring  and 
ranking  of  applicants. 


1977  STUDY  -  SELECTION  OF  AIR  TRAFFIC  CONTROL  SPECIALISTS 


Continuing  concern  with  the  rate  of  ATC  trainee  attrition  and  considera¬ 
tion  of  reestablishing  centralized  ATC  training  at  the  FAA  Academy 
resulted  in  a  review  of  FAA  selection  and  screening  policies  in 
December  1979  1  1  and  completion  of  a  cost  analysis  study  of  alterna¬ 
tives  i'"  in  March  1975.  Among  the  actions  resulting  from  this  review, 
FAA  contracted  with  Education  and  Public  Affairs,  Inc.,  (EPA'  for  a 
follow-on  analysis  of  ATC  selection  tests  in  Jui.?  1975.  A  summary  of 
the  final  EFA  report  is  provided  elsewhere  1  '  ■ 1  . 


Objectives.  The  objectives  of  this  research  were  directed  to  essentially 
the  same  concerns  as  the  previous  (19”2)  study  completed  by  EPA:  selection, 
placement,  and  fairness.  However,  the  study  design  significantly  expanded 
the  sample  size,  structure,  and  representation;  encompassed  mere  criterion 
measures  of  ATC  job  success;  and  incorporated  an  evaluation  of  prior 
aviation-related  experience  and  educational  level  as  predictors  of  ATC 
job  success. 

Longitudinal  Analysis  of  1972  Experimental  Tests.  The  first  effort  in 
this  study  was  to  analyze  on  a  longitudinal  basis  the  relationships 
between  the  experimental  test  results  obtained  during  19"1  and  criterion 
measures  related  to  the  ATC  specialists  who  participated  as  of  1975. 

Validity  coefficients  of  each  test  score  with  each  criterion  measure  were 
calculated.  A  factor  analysis  of  the  tests  was  also  undertaken  to  deter¬ 
mine  overlapping  of  underlying  variables.  The  tests  selected  from  these 
analyses  were  then  subjected  to  multiple  regression  analysis  to  determine 
the  minimum  number  of  tests  or  test  scores  which  would  predict  the 
maximum  proportion  of  variance  in  each  of  the  criterion  measures. 

Table  2  identifies  those  1971  experimental  tests  and  predictors  (X) 
which,  based  on  their  simple  correlation  with  the  various  criterion  measures 
established  for  the  analysis,  warranted  further  study  by  means  of  field 
validation. 


* 


Table 


2  Predictor  and  Criterion  Measures  of  ATC  "Success" 
J1975)  for  Controllers  Participating  in  1971  AT C 

Research 


Sepa¬ 

ration 

W“ 


Progress  ior.+ 
Attrition 


Present 
Option 
t)EV  FPL 


1971  Supv. 
Assess. 


Sup/Staff 

Position 

IE®- 


CODE  X 

Dial  Heading  X 

Dir.  Heading  X 

Air  Traffic  Prob.* 
Arith.  Reasoning** 

ATC  General  Info.  X 

Concept  Adjective  X 

Biographical  Info .  X 


X  X 
X 


X 

X 


XX  X 

-  -  X 

XX  X 

-  X  X 

X  -  X 

XX  X 


X 


*Air  Traffic  Problems  CSC  test  No.  540 

♦♦Arithmetic  Reasoning  was  Part  5  of  the  Dailey  Technical  and 
Scholastic  Test  (TST) 

♦♦♦Supervisory  assessment  criterion  data  available  for  1971  FPL 
ATCS  only.  No  new  1971  appointees  progressed  to  ATC  super¬ 
visory  or  staff  positions  by  1975. 


With  some  modifications  these  experimental  tests  used  in  1971  became  the 
"core"  test  battery  for  the  1977  research.  A  more  detailed  discussion 
of  the  methodology  and  results  of  this  longitudinal  study  is  provided 

elsewhere  (14). 


Sample  Description  -  1977  Study.  A  comprehensive  sample  design  was 
constructed  to  define  the  ATC  population  to  ensure  a  representative 
sample  for  three  specific  "year  of  hire"  groups.  These  groups  repre¬ 
sented  three  ATC  career  "stages":  (1)  New  Hires  (1976);  (2)  Develop¬ 
mental  ATC  Specialists  with  2  to  3  years  ATC  experience  with  FAA  ( 1 9  73 
and  1974);  and,  (3)  ATC  Specialists  with  2  to  6  years  experience  at  the 
Journeyman  (FPL)  level  (1969  and  1970). 

In  addition  to  these  primary  ATCS  samples,  three  additional  ATCs 
samples  were  included: 


.  An  oversample  of  currently  employed  women  and  minority 
ATCS's  in  the  same  three  "year  of  hire"  groups. 
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ATC  Specialists  who  participated  in  the  1972  research. 


.  A  sample  of  ATC  specialists  who  were  hired  during  the  three 
time  periods  sampled,  but  who  had  separated  from  ATC  work 
before  reaching  FPL  status. 


Sample  selection  was  constrained  to  exclude  employees  who  were  in  ATC 
staff  or  supervisory  positions  or  over  age  31  at  the  time  they  were 
hired  except  in  the  FSS  option. 


Sample  selection  for  the  three  primary  groups  was  based  on  stratified 
random  sampling  methods  to  provide  a  proportionally  representative  group 
of  the  total  constrained  ATC  universe  for  each  of  the  four  ATC  "options" 
(FSS,  VFR,  IFR,  and  ARTCC)  with  respect  to  both  the  initial  and  current 
option  of  assignment.  ’ 


Table  3  identifies  the  various  ATC  specialists  samples  by  year  group,  the 
total  number  desired  and  the  samples  actually  obtained.  Since  participation  was 
voluntary,  the  number  invited  was  expanded  where  possible  to  provide  for 
declination  and  nonresponses. 


TABLE  i-  ATC  SPECIALIST  BY  YEAR  HIRED, 
SAMPLE  GROUPS  AND  ATC  CAREER  STATUS 


ATC  Sample 

ATC 

Percent  of 

Year 

Year 

Career 

Number 

Desired 

Obtained 

Desired 

Hired 

Groups 

Status 

Invited 

Sample 

Sample 

Sample 

1969-70 

(1)  Employed  ATCS 

m. 

1344 

800 

754 

(94%) 

(2)  ATCS  Oversample 

FPL 

151 

200 

31 

(  6%) 

(3)  Separated  ATCS 

DEV 

— 

— 

362 

— 

1973-74 

(1)  Employed  ATCS 

DEV 

112? 

800 

740 

(93%) 

(2)  ATCS  Oversample 

DEV 

258 

200 

72 

(36%) 

(3)  Separated  ATCS 

DEV 

— 

- — 

166 

— 

1976 

(1)  Employed  ATCS 

New  Hires 

(DEV) 

610 

610 

590 

(97%) 

1971 

(1)  Employed  ATCS 

FPL 

480 

480 

270 

(56%) 

(2)  Separated  ATCS 

FPL/DEV 

— 

— 

74 

— 
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The  number  of  women  and  minorities  who  volunteered  was  not  adequate  for 
analysis  by  year  group  or  ATC  option.  Consequently,  the  analysis  on  test 
fairness  combined  year  and  option  groups  to  provide  235  women  and  321 
minorities.  The  oversample  for  women  and  minorities  was  used  only  for 
analysis  of  the  fairness  of  the  ATC  "success"  predictors.  The  final 
EPA  report  (i:|,..pu-r  vil  ,.j  t,  :  i-i  vnr,  I.)  i.;,..:  .  i  ,  ,.i  t,  ,,, 

1  -')•:!  1  ' ]  1  ,>>•.%  . 


Predictors.  Two  of  the  tests  used  in  the  1972  study  were  modified  for 
use  in  the  1977  research. 


.  CODE  (Controller  Decision  Evaluation).  This  test  consisted  of 
three  film  versions  of  a  computer  simulation  of  moving  air 
traffic  patterns  appearing  on  a  radar  scope.  Initially  these 
were  converted  to  slide  projector  presentation  to  eliminate  the 
need  for  movie  projector  equipment  and  to  simplify  both  the 
response  recording  and  scoring.  Group  administration  of  the 
various  adaptions  of  CODE  during  this  test  development  phase 
clearly  pointed  up  the  practical  problems  of  using  film  or  slide 
projection  equipment  in  test  administration.  Consequently .  a 
paper-pencil  version  was  developed  which  incorporated  measures 
of  abilities  to  identify  potential  conflicts  of  aircraft  as  well 
as  the  traditional  kinds  of  aptitudes  within  an  air  traffic  control 
context.  The  resulting  test,  the  Multiplex  Controller  Aptitude 
Test  ( MCAT )  was  made  available  to  EPA  as  a  substitute  for  the 
CODE  tests. 

.  Arithmetic  Reasoning.  In  the  prior  research,  the  arithmetic  reasoning 
test  was  one  part  of  the  Dailey  Technical  and  Scholastic  Test  (TST). 
Since  this  part  could  not  be  given  as  an  isolated  test,  an  Arithmetic 
Reasoning  test  developed  by  the  Array  Air  Force,  which  was  similar, 
was  used  in  the  1977  research. 


In  addition  to  the  predictors  derived  from  the  longitudinal  analysis  of 
the  experimental  tests  administered  in  1971,  three  other  predictors  were 
used. 


.  Pre-employment  Experience  Questionnaire  (PEQ) 

To  obtain  specific  data  from  participating  ATC  specialists  on 
various  kinds  of  pre-FAA  experience  and  education,  a  questionnaire 
was  developed  from  the  CSC  Rating  Guide  elements  used  as  a  basis 
for  granting  additional  credit  in  the  ATC  employment  selection 
process . 
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.  ATC  Occupational  Knowledge  Test  (OKT) 


This  test  was  developed  to  be  "job-knowledge  specific."  Conse¬ 
quently,  it  was  not  included  in  the  1977  research  for  the  purpose 
of  evaluating  its  use  in  screening  ATC  job  applicants  for  employ¬ 
ment  eligibility.  It  was  used  to  measure  the  "quality"  of  prior 
experience  as  a  potentially  improved  basis  for  granting  additional 
credit  for  experience  in  the  selection  process  in  place  of  the 
existing  CSC  Rating  Guide. 

.  Sixteen  Personality  Factor  Questionnaire  (16  PF) 

The  16  PF  Questionnaire  is  administered  as  part  of  the  medical 
qualification  process  to  all  entering  ATC  Specialists.  It  is  de¬ 
signed  to  measure  important  personality  characteristics  not  other¬ 
wise  measured  and  the  1977  research  offered  the  opportunity  to  assess 
its  utility  for  possible  selection  or  placement  purposes. 


The  experimental  test  battery  used  in  the  1977  research  may  be  divided 
roughly  into  two  groups  of  instruments;  (1)  Cognitive  Predictors,  tests, 
which  have  predetermined  right  or  wrong  answers;  and  (2)  Other  Predictors 
where  applicant  responses  ordinarily  have  no  inherent  value  of  "right"  or 
"wrong"  except  as  measurable  against  criteria  or  values  external  to  the 
"test."  The  following  identifies  the  predictors  used  within  these  two  groups. 

.  Cognitive  Predictors 

Multiplex  Controller  Aptitude  Test  (MCAT) 

Directional  Headings  Test 
Dial  Reading  Test 
Arithmetic  Reasoning  Test 
ATC  General  Information  Test 
Present  CSC  Test  Battery  (5  parts) 

ATC  Occupational  Knowledge  Test 

.  Other  Predictors 


Pre-Employment  Experience  Questionnaire 
Concept  Adjective 
Biographical  Inventory 

Sixteen  Personality  Factor  Questionnaire  (16  PF) 
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Rather  than  re-administer  the  present  CSC  test  battery,  the  intent  was  to 
obtain  the  CSC  test  scores  for  participating  ATC  specialists  from  existing 
records.  However,  this  did  not  prove  to  be  feasible  for  a  large  number  of 
the  employees  in  the  sample.  Consequently,  analysis  of  the  experimental 
tests  in  relation  to  the  existing  CSC  test  battery  as  predictors  of  ATC 
success  was  not  possible  in  this  study. 

The  "cognitive  predictors"  and  "other  predictors"  were  each  analyzed 
separately  against  the  criterion  measures  used  to  define  "success"  in  the 
ATC  occupation.  They  were  also  analyzed  together  in  a  supplemental  report 
0  5)  to  determine  the  extent  to  which  "other  predictors"  added  to  the 
validity  of  the  "cognitive  predictors." 

A  discussion  and  description  of  each  of  the  predictors  used  in  the  study 
including  relevant  prior  research,  reliability  and  validity  is  provided 
elsewhere  (Chapter  VI  or  reference  14). 


For  analytic  purposes,  the  objective  was  to  obtain  pre-employment  and 
education  information  on  all  ATC  specialists  in  the  sample  groups.  The 
experimental  test  battery  was  administered  to  approximately  50  percent 
of  the  1969-1970  and  1973-1974  sample  and  all  new  hires  in  the  1976  group. 
Selection  of  employees  to  take  the  experimental  test  battery  was  controlled 
to  provide  proportional  distribution  by  ATC  option  within  each  year  group. 

The  experimental  tests  were  also  administered  to  all  employees  included  in 
the  ATCS  oversample  groups. 

Predictor  data  were  obtained  in  several  ways.  The  Prior  Experience  Question¬ 
naire  (PEQ)  was  completed  by  the  ATC  specialists  at  the  time  they  agreed  to 
participate.  The  experimental  tests  were  administered  to  the  new  hires 
at  the  FAA  Academy  on  their  first  day  of  attendance  by  the  EPA  staff 
assisted  by  FAA  personnel.  For  those  ATC  specialists  assigned  to  facilities 
the  tests  were  given  by  FAA  test  administrators  who  were  trained  by  EPA. 

A  full  discussion  of  the  data  collection  methodology  is  provided  in  the  FAA 
report  (Chapter  V! II  of  reference  14). 

Criteria  and  Measures  of  ATC  Success.  To  determine  how  valid  experimental 
tests,  prior  experience  and  education  are  as  predictors  of  ATC  success, 
operational  definitions  (criteria)  of  success  and  ways  to  quantitatively 
measure  them  had  to  be  established.  The  following  four  criterion  measures 
were  used  in  the  study  and  were  then  combined  into  a  single  "aggregate" 
measure  of  ATC  success. 
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Training  Performance.  This  was  measured  by  scores  received  on 
the  ATC  Laboratory  Problems  and  the  Controller  Skills  Test  during 
initial  ATC  training  at  the  FAA  Academy.  These  scores  were  selected 
since  they  require  students  to  demonstrate  operational  application 
of  academic  knowledge. 

On-the-job  Performance.  This  was  measured  by  confidential  job/task 
assessments  prepared  by  the  employee's  supervisor.  It  included 
54  questions  on  ATC  job  tasks  and  four  general  questions  on  quality 
of  job  performance.  Responses  to  a  seven-point  "overall"  rating 
scale  were  selected  as  the  measure  for  on-the-job  performance. 

Progression.  This  was  measured  by  the  ATC  "option"  to  which  the 

ATCS  was  initially  assigned  when  hired,  compared  to  the  "option" 
the  ATCS  was  assigned  to  on  January  1,  1976. 

Four  "options"  were  defined  to  represent  general  complexity  levels 
of  ATCS  work  (FSS,  VFR,  IFR,  ARTCC)  with  ARTCC  defined  as  tne 
"highest"  complexity  level.  Within  this  hierarchy,  progression 
values  were  established  as  follows: 


.  "High" — assigned  when  an  ATCS  was  in  an  option  of  a  com¬ 
plexity  level  the  same  as  or  higher  than  the  initial  option 
to  which  assigned  (i.e.,  VFR  initial  assignment;  IFR  current 
assignment  on  January  1,  1976). 


.  "Low" — asig.ned  when  an  ATCS  was  in  an  option  of  a  lower 
complexity  level  than  the  option  to  which  initially 
assigned  (i.e.,  ARTCC  initial  assignment;  fss  current 
assignment  on  January  1,  1976). 


Attrition.  This  was  measured  by  whether  or  not  ATC  Specialists  hired 
during  the  year  groups  sampled  for  this  study  were  still  employed  in 
ATC  work.  Those  still  employed  as  ATC  Specialists  were  assigned  a 
"high"  score;  those  separated  were  assiqned  a  "low"  score. 

Aggregate  Criterion  of  ATC  Success.  This  was  constructed  from  the 
four  individual  criterion  (training,  on-the-job  performance  progres¬ 
sion  and  attrition)  and  their  measures  and  provides  a  five-point 
scale  value  for  ATC  "success." 


The  predictors  of  ATC  "success"  used  in  this  study  —  experimental 
tests,  prior  experience,  and  education  —  were  analyzed  for  validity 
against  each  of  the  four  criterion  measures  of  ATC'success"  —  training 
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scores,  supervisory  assessments,  progression,  and  attrition.  Final 
conclusions  and  recommendations  are  based  on  the  validity  of  the 
predictors  with  the  aggregate  criterion  of  ATC  success.  A  full 
discussion  of  the  development  of  each  of  the  criterion  measures  is 
provided  in  the  EPA  Report  irbupui  v  n  •  ^jvn.  t-  j.. >. 

Descriptive  Information.  Data  on  the  education  and  experience  background 
of  ATC  specialists  who  participated  in  the  1977  study  are  Drcvided  in  the 

KPA  Report  (Chapter  VI  1  o!  rel'erenee  J  .1  .  .  .  - 1  :•  ..  >•.<.  tvi  >. 

the  three  "year-el  -hire"  jtroups  (aiui  t  in  1(*,'  _  v  v-;  ,•  eh.^eti  u.,. 

level,  military  service  and  pilot  experier.ee.  .  I,  .  ,  .  l:.l  ..-.ri 

sample  croups  on  selected  variables. 


TABLE  4.  EDUCATION  ARP  EXPERIENCE  LEVEL? 
Year  Hired  as  ATCS 

1969—0 


Prior  to 
1971 

1969 

197C 

19~3 

1974 

1973-74 

Oversar.ple 

1976 

(N=304) 

(N=659 ) 

(N=6fcl > 

(N*1C3) 

(N=592 

High  School 

72% 

34% 

24% 

25% 

16% 

Some  College 

25% 

53% 

53% 

59% 

56% 

College  Degree (s) 

1% 

13% 

23% 

16% 

2  ?% 

Military  Service 

96% 

75% 

74% 

56% 

71% 

ATC  Experience 

72% 

— 

— 

— 

— 

IFR 

— 

35% 

39% 

27% 

32% 

VFR 

— 

39% 

38% 

26% 

37% 

Pilot 

2% 

30% 

34% 

11% 

■33% 

The  sample  of  304  ATCS  hired  prior  to  1971  was  derived  from  the  1972  study  ( 
ATC  Specialists  could  have  been  hired  between  i960  and  1970. 

Analytical  Methodology.  Before  undertaking  the  validity  analysis,  a  number 
of  preliminary  analyses  were  made  to: 

.  Substantiate  that  the  sample  of  ATC  specialists  who  did  volunteer 
was  not,  except  for  the  proportion  of  minorities,  essentially 
different  from  those  who  did  not  volunteer. 


.  Calculate  the  means,  standard  deviations,  reliability,  and  inter¬ 
correlations  of  experimental  predictors  and  ATC  success  criterion 
measures. 
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.  Determine  that  the  order  of  MCAT  test  administration  did  net  affect 
test  results. 

.  Analyze  training  scores  and  convert  them  to  standardized ' Z )  scores. 

.  Analyze  pre-employment  experience  and  establish  empirical  scale 
values  for  validation. 

.  Analyze  predictors  for  inclusion  in  the  validation  study. 

A  full  discussion  of  these  preliminary  analyses  is  prevideo 

report  1 1  ii.r.  U'i  X  .  !  .  .  .■ .  :  , 

-oai'jS  ♦  si » i dr  ’.  i  ; :  i .  ; ;  -  ,  v 

thi'i'u.'ii  IX..;  .  :  i  h.:t  :-v  :  r;  =  .  -  . .  .  •  ;  : 

t'or  tile  noil-.  :\ •  v.  L;t 


As  a  result  of  these  preliminary  analyses,  two  of  the  cognitive  tests 
were  excluded  from  the  validation  analysis. 

.  Civil  Service  Tests.  The  inability  to  obtain  tr.e  scores  on  t.oe 
CSC  test  battery  for  a  significant  number  of  A7C  specialists  in 
the  sample  ana  the  effect  of  restriction  in  range  or.  those  scones  vhi 
were  available,  seriously  interfered  wot n  proceeding  with  the 
validity  analysis  of  the  CSC  tests  in  the  EPA  study. 

.  ATC  General  Information  Tests.  This  test  'was  excluded  from  validity 
analysis  since  it  was  only  administered  to  the  new  hires  attending 
the  FAA  Academy  training;  it  was  not  given  to  the  19?3-"d  or  1969-70 
ATC  samples  because  of  time  constraints  on  field  testing. 

Consequently,  the  following  predictors  were  included  in  the  primary  vali¬ 
dation  analysis: 


Multiplex  Controller  Aptitude  Test  (MCAT) 

Directional  Heading  Test 

Dial  Reading  Test 

Arithmetic  Reasoning  Test 

ATC  Occupational  Knowledge  Test  (OKT) 

Pre-employment  Experience  (PEQ) 


The  following  non-cognitive  predictors  were  included  in  the  supplemental 
validation  analysis: 
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Concept  Adjective 

Biographical  Inventory- 

Sixteen  Personality  Factor  Cuestionnaire  (16PF; 

In  summary,  the  validation  methodology  for  the  cognitive  predictors  (and 
prior  experience)  consisted  of  the  following  steps: 

1.  Statistical  analyses  of  each  of  the  smallest  homogeneous  samples 
(i.e.  each  ATC  option  and  all  ATC  options  combined  within  each 
year  group  against  each  of  the  four  individual  ATC  success 
criterion  separately).  This  resulted  in  l1?  separate  analyses. 

2.  Predictors  selected  from  Step  1  (based  or  validity  coefficients 
and  significant  levels)  for  each  ATC  option  and  ail  ATT  opticr.s 
combined  across  ail  year  groups  were  then  analyzed  to  identify 
the  best  overall  set  of  predictors  for  each  cf  the  four  individual 
criterion  measures.  This  resulted  in  I7  separate  analyses. 


3.  Results  of  each  of  the  four  individual  criterion  measures  were 
then  examined  for  each  ATC  option  and  all  options  coccined  to 
determine  the  best  set  of  predictors  across  criterion  measures. 

This  resulted  in  17  separate  analyses. 

4.  The  final  set  of  predictors  from  Step  3  were  then  validated  against 
the  aggregate  ATC  success  criterion  leading  to  derivation  cf 
weighted  test  and  experience  scores.  This  involved  five  separate 
analyses. 

5.  The  weighted  test  battery  and  experience  scores  were  then  validated 
against  the  four  single  criterion  measures.  This  involved  four 
separate  analyses. 

The  specific  analytical  techniques  applied  and  the  rationale  for  each  of 

till'  st  ips  ill  t  Ik-  :  f  t  IuMi  -  I  i'.'  uiju-. :  .  i  iv  :  u  i  !  i : j  ;•  r t  >  :  X 
nu  l-l-fllO-  1..  i  111-  results  «'t  tile  Uil.ll'.si  .Ulii  till  v  .1  I  i.l  it  it  j '  t  .1  ;  III  d  .ill 
prov  i.iod  in  Cii.ipUrs  XI,  XII.  XIII,  Xl\  und  XV,  i\  •  nviu  <  I  . ,  tin. 

Append  i  ues  L'<  I  lie  I  i  11.1  1  I  I’A  repel  l  I  I  )  . 

The  methodology  previously  discussed  was  modified  for  analyses  of  the 
non-cognitive  predictors  to  compensate  for  the  differences  imposed  by  the 
use  of  "personality"  as  opposed  to  aptitude  or  knowledge  tests. 

The  most  significant  change  in  methodology  resulted  from  the  need  to 
empirically  establish  the  appropriate  directionality  of  the  various  non- 
cognitive  predictor  scales  before  undertaking  validity  analysis.  The 
specific  analytical  techniques  applied  in  evaluating  the  non-cognitive 
predictors  are  identified  in  reference  r>. 


The  methodology  to  evaluate  the  fairness  of  the  experimental  test  batt 
prior  experience  and  the  Occupational  Knowledge  Test  was  directed  tc 
determining  differential  validity  in  accordance  with  the  Uniform  Guide 
on  Employee  Selection  Procedures  »  l >  which  state: 

"When  members  of  one  racial,  ethnic  or  sex  group  characteristics! 
obtain  lower  scores  on  a  selection  procedure  than  members  cf 
another  group,  and  the  differences  are  not  reflected  in  di fferen 
in  measures  of  job  performance,  use  of  the  selection  procedure 
may  unfairly  deny  opportunities  to  members  of  the  group  that 
obtains  the  lower  scores." 

The  methodology  for  evaluating  fairness  aspects  of  tne  predictors  use: 
in  this  research  is  discussed  in  cii.ipu-r  xv  I  .  :  » .  n  •  . 

Results.  The  following  briefly  summarizes  the  results  of  this  restart 
study.  More  detailed  discussion  is  provided  in  references  ■- 
L5 . 
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Two  of  the  experimental  tests — Multiplex  Controller  Aptitude 
and  Directional  Headings — predicted  the  ATC  success  criteria 
established  for  the  study  at  statistically  significant  levels 
confidence  for  all  ATC  options  combined  and  for  the  three  pro 
year  groups  sampled  (1969-70;  1973-74;  and  19761.  Twc  other 
perimental  tests — Arithmetic  Reasoning  and  Dial  Reading— eitr 
not  predict  the  ATC  success  criteria  or  did  net  add  appreeiat 
the  prediction  values  obtained  from  MCAT  and  Directional  Head 


Preemployment  aviation-related  experience  and  the  ATC  Occupat: :: 
Knowledge  Test,  while  not  intended  for  use  in  determining  ini:: 
appointment  eligibility,  predicted  ATC  success  at  statistically 
significant  levels  of  confidence  and  increased  the  validity  co¬ 
efficients  obtained  with  the  experimental  tests. 


Education  beyond  high  school  level  prior  to  FAA  employment  did 
predict  ATC  success  in  either  a  positive  or  negative  direction. 
Essentially  all  of  the  controllers  in  the  samples  included  in  t 
study  had  at  least  a  high  school  education. 


Other  experimental  instruments — 16  Personality  Factor  Questions: 
Concept  Adjective  and  Biographical  Information  Questionnaire — di 
add  appreciably  to  the  predictive  capability  of  the  experimental 
test  battery,  preemployment  experience  and  the  ATC  Occupational 

Knowledge  Test. 


The  validities  derived  from  the  analysis  of  combined  predictors  against 
the  Aggregate  ATC  Success  Criterion  by  ATC  option  for  all  year  groups 
combined  are  provided  in  Table  5.  Weights  for  the  experimental  tests 
were  derived  from  the  multiple  regression  analysis.  Separate  weights 
were  developed  for  each  test  for  each  ATC  option  and  for  all  ATC  options 
combined.  The  derived  weights  for  all  ATC  options  combined  were  MCAT 
Conflict — 37;  MCAT  Aptitude — 21;  Directional  Headings — 15;  and,  Dial 
Reading — 0 . 


TABLE  5.  VALIDITIES  OF  COMBINED  PREDICTOR3 
AGAINST  THE  AGGREGATE  ATC  "SUCCESS"  CRITERION 
(BY  ATC  OPTION  --  ALL  YEARS  COMBINED) 


ATC 

Weighted  Test 
Battery  Scores 

Weighted  Test 
Battery  Plus 

PEQ 

W-eighted  T est 
and  PEQ  Plus 
OKT 

Ooti  on 

N 

r 

df 

R 

df 

R 

FSS 

196 

.  23** 

193 

.  26** 

192 

.  26** 

VFR 

479 

.26** 

474 

.  44** 

423 

.45** 

IFR 

499 

.26** 

494 

.  39** 

443 

.43** 

ARTCC 

445 

.  30** 

425 

.  32** 

388 

.  37** 

All  Options 

1309 

.  26** 

1287 

.  32** 

1205 

.  34** 

**  *  p£. 0 1 


Academy  trainees  (1976)  in  the  Terminal  option  were  not  identified  by  IFR 
or  VFR  options.  Consequently,  they  were  included  in  both  the  IFR  and 
VFR  option  Ns,  but  only  counted  once  in  the  total  N's  for  all  ATC 
options. 

Reference  (13)  Appendix  A  shows  the  validity  coefficients  derived  for 
the  experimental  test  battery  (unweighted),  prior  experience,  ATC 
Occupational  Knowledge  Test  (OKT)  and  education  level  for  each  of  the 
four  ATC  success  criteria  (excluding  the  aggregate  criterion)  by  ATC  option. 
Appendix  B  (13)  shows  the  validity  of  selected  experimental  tests  with  the 
aggregate  ATC  success  criterion. 
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Placement 


.  The  weighted  experimental  test  battery  did  differentiate  between 
the  FSS,  Terminal  (IFR,  VFR)  and  ARTCC  options.  Average  scores  for 
ATC  specialists  in  FSS  terminals  and  ARTCCs  were  different  at  statis¬ 
tically  significant  levels  of  confidence.  The  average  FSS  score 
was  lowest  and  ARTCC  highest.  Table  6  provides  the  comparative 
mean  scores  for  the  various  ATC  options  and  year  groups. 


TABLE  6.  MEAN  WEIGHTED  TEST  BATTERY  SCOPES 

BY  ATC  OPTION 


1969-70  &  1973-74  1976  NEW  ATC  HIRES 


ATC  ATC  HIRES  (ACADEMY  TRAINEES) 


Option 

N 

Mean 

SD 

N 

Mean 

SD 

FSS 

196 

227.4 

40.2 

VFR 

170 

244.6 

35. 8} 

(?-E-R^)no 

242.0 

35.0 

IFR 

189 

250.0 

37.3] 

ARTCC 

182 

264.3 

36.6 

263 

247.9 

38.4 

Analysis  of  variance  shows  that  the  means  of  scores  differ  significantly 
at  the  1  percent  level  of  confidence  between  FSS,  Terminal  (VFR  and  IFR) 
and  ARTCC  for  the  1969-70  and  1973-7^  groups.  The  1976  A'i c  ^ruup 
difference  between  terminal  and  ARTCC  i s  significant  at  the  5  percent  lev. 

Test  Fairness 


.  Women  as  a  group  scored  lower  on  each  of  the  predictors  and  on  the 
aggregate  criterion  of  ATC  success.  In  each  case,  these  differences 
were  significant  at  the  1  percent  level  of  confidence  (Table  7). 


TABLE  7  -  MEANS,  STANDARD  DEVIATIONS  AND  t_-TESl 
“RESULTS  FOR  PREDICTORS  AND  JHE  ACCREt-Al.E 
ATC '"SUCCESS"  'CRITERION  FOR  MEN  AND  WOMEN 


Weighted 

Test  Battery 

-  -  Predictors-  - 

PEQ 

-  -  - 

orr 

- - 

-  -Criterion  -  - 
Aggregate  ATC 
"Success" 

Sex 

N  Mean  SD 

N  Mean  SD 

N 

Mean 

SD 

N  Mean  SD 

Men 

1397  244.5  380 

2736  3.7  2,9 

1318 

76.6 

12.3 

1254  3.1  1.4 

Honan 

171  234.5  480 

235  .8  1,6 

158 

64.6 

16.3 

158  2.6  1.4 

(t»3 . 14 / p<. 01) 

(t-15.02;p<.01) 

(t-11 

.21;  p<.  01) 

(t“4.56;  p<.01 
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Validities  of  the  predictors  against  the  aggregate  ATC  success 
criterion  are  provided  in  Table  8.  Except  for  prior  experience, 
these  validities  are  significant  at  the  1  percent  or  5  percent  levels. 


TABLE  8  -VALIDITIES  OF  THE  WEIGHTED  TEST 
BATTERY,  PEQ  AND  OKT  AGAINST  THE 
AGGREGATE  ATC  "SUCCESS"  CRITERION  BY  SEX 
(*«p<.05;  **«p<.01) 


Weighted 
Test  Battery 

Sex  N  r 

Men  1386  .23** 

Women  165  .19** 


Prior  Experi- 
ence  (PEQ) 

_ N_  r 

2721  .21** 

229  -.04 


ATC  Occupational 
Knowledge  (OKT) 

_ N_  '  r 

1308  .22** 

154  .14* 


It  should  be  noted  that  few  of  the  229  women  in  the  prior  experience 
sample  had  aviation-related  experience.  Consequently,  the  validity 
results  are  due  to  lack  of  variance. 


.  Comparable  statistical  analyses  were  made  between  minority  and  non- 
minority  groups  and  between  non-minority  and  blacks  as  defined  by 
0MB  Circular  A-46.  The  results  with  respect  to  the  test  battery, 
prior  experience  and  the  ATC  Occupational  Knowledge  Test  are 
provided  in  Tables  9,  10  and  11  respectively. 
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A. 


Non-Minorities 
All  Minorities 


Non-Minorities 

Blacks 


Non-Minorities 
All  Minorities 


Non-Minor it ies 
Blacks 


Non-Minorities 
All  Minorities 

Non-Minorit ies 
Blacks 


TABLE  9  -  IKS!  SAM!'!.! 


PREDICTOR 


CRITERION 


We 

i s’li ted  T 

est 

Ac  p  rec.at  e 

ATC 

Bat t  cry 

H 

Success 

1  t 

N 

Mean 

sn 

N 

Mean 

SD 

1323 

247.7 

371 

1  308 

1.2 

1  .4 

245 

220.4 

479 

24  3 

2 .  / 

!  .4 

(t  = 

10. 30: p 

.01  ) 

( t  -  5 .  (19 ; 

5 . .’ 

.01) 

1  323 

247.7 

171 

1  508 

1  .4 

145 

207.4 

407 

144 

■i  / 

1  .  5 

(t  = 

12 . 30 :  p 

.01) 

<  t  =(■>.  3  1  :  p 

.01) 

TABLE  10  -  PRIOR  1 

EXPERIENCE  SAMPLE 

PREDICTOR 

CRITERION 

Prior  Aviation 

Aggregate  Alt' 

Experience  (PEQ) 

»»..  II 

>  I  K'  1*0 SS 

N  Mean  SD 

N  Mean 

SD 

2115  3.9  2.9 

2097  5.2 

1  .4 

J121 _ 2._4 _ 3,0 

318  2.6 

1  .4 

(t=8.48:p  .01) 

( t  -  7  .  1  5 :  p 

.01) 

2115  3.9  2.9 

2097  3.2 

1.4 

194  1.7  2.8 

193  2.4 

1  .  1 

( t  = 1 0 . 27 :p  .01) 

(t  =  8. 3  9 : p 

.01 ) 

TABLE  II  -  ATC  OCCUPATIONAL  KNOUT  .EDGE  SAMPLE 


PREDICTOR 
ATC  Occupational 


CRITERION 
Approbate  ATC 


Knowledge  Test  "Success" 


N 

Mean 

SD 

N 

Mean 

SD 

1247 

76.5 

12.8 

1  235 

5.  2 

1  .4 

229 

69.1 

14.7 

22  7 

2.7 

1 .4 

(t=7 

.  85  :p- 

-  .01) 

(1  = 

=  4  .  36 :  p 

.01) 

1247 

"76.5 

12.8 

1  2  55 

5.2 

1  .4 

134 

66.9 

15.2 

1  55 

2.  5 

1  .  5 

OO 

II 

4J 

•  1 2 :  p 

-01 )  ' 

(t  = 

=8. 1 2:p 

.01) 
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Minorities  as  a  group,  and  blacks  as  an  element  of  the  minority 
group,  scored  lower  on  each  of  the  predictors  and  on  the  aggre¬ 
gate  ATC  success  criterion.  In  each  case,  these  differences  were 
significant  at  the  1  percent  level  of  confidence. 

Validity  of  each  of  the  predictors  with  the  aggregate  success 
criterion  for  minorities  and  non-minorities  are  provided  in 
Table '12. 


Correlations  for  th»  weighted  teat  batterv  PEP 
acale  and  OKT  against  the  aggregate  criterion 
by  ainoritv/non-alnorl ty  status  ~~ 


Table  12 


Non-minority 

Minority 


Weighted 
Test  Batterv 

37 — ^ 

1308  203** 

243  .219** 


2097  . 148** 
318  .247** 


1233  .201** 

227  .272** 


When  blacks  were  analyzed  separately  from  all  other  minorities, 
the  validity  of  prior  experience  and  the  Occupational  Knowledge 
Test  were  sustained  (r  =  .259  and  .256,  both  at  the  1  percent 
level  of  confidence).  Validity  of  the  weighted  test  battery  was 
.120  which  did  not  reach  the  5  percent  level  of  confidence 
(p  =  .075).  However,  the  difference  in  the  validity  for  non-minori¬ 
ties  and  blacks  was  not  statistically  significant. 


Conclusions 


The  results  of  this  study  supported  the  conclusion  that  several  of  the 
experimental  predictors  were  sufficiently  valid  and  fair  in  predicting 
ATC  success  that  they  should  be  further  developed  for  operational  use 
in  the  selection  of  applicants  for  the  air  traffic  control  occupations. 


However,  the  lack  of  sufficient  CSC  test  scores  for  the  sample  popu¬ 
lation  and  the  restriction  in  range  on  those  scores  which  were  availa¬ 
ble  made  it  impractical  to  compare  the  validity  of  the  current  CSC  test 
battery  with  the  validity  of  the  experimental  tests.  In  order  to  address 
these  questions,  the  Civil  Service  Commission,  new  the  Office  of 
Personnel  Management  (0PM),  and  FAA  jointly  directed  efforts  to  obtain 
the  necessary  information  from  a  group  of  about  11,500  applicants  for 
ATC  work  during  the  Fall  of  1976  and  Spring  of  1977. 
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STUDY  OF  ATC  JOB  APPLICANTS  1976-1977 


Objectives.  Historically,  there  has  been  little  or  no  information  availa¬ 
ble  on  the  characteristics  of  individuals  who  apply  for  positions  in  the 
Air  Traffic  Control  occupation.  Consequently,  this  study  was  developed 
to  collect  and  analyze  data  on  ATC  applicants  with  respect  to  sex,  edu¬ 
cation,  prior  aviation-related  experience  and  the  relationship  between 
these  variables  and  scores  on  the  CSC  test  battery  and  experimental  tests. 
One  direct  use  of  this  information  was  to  establish  a  basis  for  comparing 
the  validities  of  the  CSC  test  and  the  experimental  tests  by  correcting 
for  the  restriction  in  range  on  CSC  test  scores  for  approximately  2,000 
newly  hired  ATC  trainees  who  attended  the  FAA  Academy  during  1976-1977  and 
for  whom  CSC  test  scores  were  available. 


Sample  Description.  During  the  period  November  1976  through  January  1977, 
the  Civil  Service  Commission  opened  the  competitive  ATC  examination 
in  the  FAA  Eastern  and  Southern  regions.  These  included  the  areas  en¬ 
compassed  by  Regions  II,  III  and  IV  of  the  Standard  Federal  Regions. 
Approximately  11,500  applicants  took  the  ATC  Civil  Service  test  during 
this  period.  Of  this  group,  about  7,500  also  completed  the  experimental 
tests  and  the  Prior  Experience  Questionnaire.  While  the  sample  from  the 
11,500  population  could  not  be  controlled,  differences  between  the  means, 
standard  deviations  and  other  statistics  on  the  CSC  tests  for  the  4,000 
who  did  not  complete  the  experimental  tests  or  provide  prior  experience 
information  were  not  statistically  significant  from  those  who  did. 

Predictors .  In  addition  to  the  five-part  CSC  test  battery,  two  experi¬ 
mental  tests,  MCAT  706A  and  the  Directional  Headings  Test,  were  administered 
to  the  applicant  group.  These  test  forms  were  the  same  versions  as  used 
in  the  1977  study  conducted  by  Education  and  Public  Affairs  "  ;  ■  : 

reference  14).  The  same  Prior  I  xper  jiaiee  iennai  re  <  I  ii  .i !  .-hit). 

All  tests  were  administered  by  CSC  examiners  as  part  of  the  normal  com¬ 
petitive  testing  procedures.  Applicants  were  informed  that  the  experi¬ 
mental  tests  would  have  no  bearing  on  their  eligibility  status  and  the 
experimental  predictors  were  administered  after  completion  of  the  competi¬ 
tive  CSC  test.  The  Directional  Headings  Test  and  the  Prior  Experience 
Questionnaire  had  to  be  manually  scored  and  converted  to  ADP  records. 

This  was  done  by  FAA  under  contract.  Determinations  on  whether  appli¬ 
cants  were  men  or  women  were  coded  on  the  Prior  Experience  Questionnaire 
based  on  the  applicants  name.  At  this  point,  Federal  regulations  pro¬ 
hibited  obtaining  any  ethnic  or  minority  group  information  from  the 
applicants. 
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Criteria.  Since  this  study  encompassed  ATC  applicants,  it  was  not  feasible 
to  establish  operational  criterion  measures  of  validity.  However,  a 
variety  of  statistical  analyses  were  made  based  on  men  and  women  appli¬ 
cants  and  those  who  passed  the  CSC  test  in  contrast  to  those  who 
failed. 


Analytical  Methodology.  The  data  obtained  on  this  ATC  applicant  group 
were  analyzed  with  respect  to  frequency  distributions,  means,  standard 
deviations,  analysis  of  variance  and  intercorrelation  of  predictors. 


Results.  Table  13  shows  the  means  and  standard  deviations  for  the  total 
applicant  group,  the  pass  group  (those  who  scored  above  209  on  the  CSC 
test),  and  the  fail  group  by  men  and  women  applicants. 

TABLE  13 


SELECTION  TEST  PERFORMANCES,  APPLICANT  GROUPS 


CSC  TESTS 


TEST 

TOTAL  GROUP 

PASS  GROUP 

FAIL  GROUP 

SEX 

(W) 

MEAN 

SD 

(W) 

MFAN 

SD 

M 

MEAN 

SD 

CSC-24 

Total 

(7412) 

39.66 

9.6 

(3960) 

44.80 

6.8 

(3722) 

34.61 

9.3 

Men 

(5720) 

39.63 

9.5 

(2980) 

44.51** 

6.9 

(2740) 

34.32** 

9.2 

Women 

(1607) 

39.83 

9.7 

(663) 

45.91 

6.6 

(944) 

35.56 

9.  3 

CSC-51 

Total 

(7412) 

26.65 

6.9 

(3960) 

30.74 

3.8 

(3722) 

22.60 

6.4 

Men 

(5720) 

27.40 

6.4 

(2980) 

31.05 

3.6 

(2740) 

23.43 

6 . 4 

Women 

(1607) 

23.98** 

7.0 

(663) 

29.37** 

4.4 

(944) 

20.20** 

6.0 

CSC-135* 

Total 

(7412) 

21.49 

8.6 

(3960) 

27.00 

5.1 

(3722) 

16.03 

7.8 

Men 

(5720) 

21.88 

8.4 

(2980) 

26.97 

5.1 

(2740) 

16.35 

7.8 

Women 

(1607) 

20.13** 

9.0 

(663) 

27.11 

5.0 

(944) 

15.23** 

7.9 

CSC- 15 7* 

Total 

(7412) 

29.29 

10.3 

(3960) 

36.65 

b.O 

(3722) 

21.99 

8.  3 

Men 

(5720) 

29.47 

10.2 

(2980) 

36.40** 

6.0 

(2740) 

21.93 

8.2 

Women 

(1607) 

28.62** 

10.9 

(663) 

37.75 

5.8 

(944) 

22.21 

8.0 

CSC-540 

Total 

(7412) 

28.98 

13.1 

(3960) 

37.  70 

10.1 

(3722) 

20.34 

9.6 

Men 

(5720) 

29.78 

12.8 

(2980) 

37.84 

9.9 

(2740) 

21.03 

9.5 

Women 

(1607) 

26.11** 

13.7 

(663) 

37.00 

10.8 

(944) 

18.46** 

9.8 

CSC  TOTAL 

Total 

(7412) 

202.03 

52.3 

(3960) 

244.26 

23.2 

(3722) 

160.16 

37.3 

Men 

(5720) 

205.04 

50.9 

(2980) 

244.23 

22.9 

(2740) 

362.42 

36.8 

Women 

(1607) 

191.20** 

55.2 

(663) 

244.26 

24.0 

(944) 

154.08** 

37.8 

EXPERIMENTAL  TEST? 

MCAT  (CSC- 

510) 

Total 

(6822) 

25.76 

9.1 

(3340) 

31.65 

7.2 

(3481) 

20.12 

6.  9 

Men 

(5241) 

26.57 

9.0 

(2688) 

32.14 

7.0 

(2552) 

20.72 

7.0 

Women 

(1498) 

22.95** 

8.7 

(607) 

29.53 

7.2** 

(891) 

18.46** 

6.  5 

DIR  HEADINC 

Total 

(7073) 

46.99 

22.6 

(3583) 

59.93 

17.3 

(3489) 

33.68 

19.4 

Men 

(5462) 

48.87 

21.9 

(2899) 

60.63 

16.6 

(2562) 

35.56 

10.5 

Women 

(1526) 

40.17** 

23.3 

(637) 

56.58 

19.4** 

(889) 

28.41** 

18.2 

*CSC  135  (following  oral  directions)  and  CSC  157  (Abstract  Reasoning)  tests  are  doubled 
weighted  in  computing  the  CSC  teat  total  score 


**Signif leant  at  the  1  percent  level  of  confidence. 
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From  Table  13  it  is  evident  that  in  total,  women  score  somewhat  below 
men  on  most  of  the  CSC  tests  and  on  the  two  experimental  tests.  These 
differences,  while  statistically  significant,  are  not  statistically 
important  due  to  the  large  sample  size.  For  example,  further  analysis 
shows  that  sex  accounts  for  only  1  percent  of  the  variance  in  the  CSC 
total  test  score;  3  percent  in  the  MCAT;  and  2  percent  in  the  Directional 
Headings  total  test  scorjs.  In  other  words,  most  of  the  variance  in  these 
three  predictors  is  due  to  factors  other  than  sex. 


In  terms  of  the  "pass"  group,  the  mean  CSC  test  score  for  men  and  women 
is  essentially  the  same.  Consequently,  discounting  veterans  preference 
and  additional  credit  allowed  for  aviation-related  experience,  women  who 
pass  the  CSC  test  should  have  about  the  same  opportunity  to  be  selected 
for  the  ATC  occupation  as  men. 


Frequency  distribution  for  each  of  the  five  CSC  tests  and  the  two  experi¬ 
mental  tests  were  also  examined.  Two  of  the  CSC  tests,  CSC  135  and  CSC  51, 
show  a  marked  negative  skew  (-1.30  and  —  1 . 80 )  respectively)  and  conse¬ 
quently,  provide  little  differentiation  between  the  applicants. 


FIGURE  2 


CSC  -  FOLLOWING  ORAL  DIRECTIONS  CSC  -  SPATIAL  RELATIONS 

135  (DOUBLE  WEIGHTED)  51 
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Frequency  distribution  graphs  of  all  of  the  CSC  and  experimental  tests  are 
provided  in  reference  -0- 

Table  14  provides  descriptive  information  on  education  and  experience  levels 
for  men  and  women  in  the  applicant  group.  Aviation-related  experience  data 
was  obtained  from  the  Prior  Experience  Questionnaire  (PEQ).  This  identified 
the  specific  work  or  skill  elements  which,  based  on  the  present  CSC  feting, 
allows  granting  extra  credit  points  to  those  applicants  who  pass  the  CSC  test. 

TABLE  14 


Descriptive  Statistics-Applicant  Group 


EDUCATION  LEVEL 


TOTAL  APPLICANTS 


Men  with 


High  school  or  less 
Less  than  3  years  college 
3  or  more  years  college 


1225  (212) 
2352  (411) 
2126  (372). 


Total 


5703  (1002) 


Women  with: 

High  school  orless 
Less  than  3  years  college 
3  or  more  years  college 


3S5  (222) 
618  (392) 
630  (392) 


Total 


1603  (1002) 


EXPERIENCE  LEVEL 


Men  with 


No  aviation-related  experience 
Aviation-related  experience 

Total 

Women  with: 

No  aviation-related  experience 
Aviation-related  experience 

Total 

EDUCATION  AND  EXPERIENCE 

With  aviation-related  experience 

High  achool  or  less 
Leas  than  3  years  college 
3  or  more  years  college 

No  aviation-related  experience 

High  achool  or  less 
Less  than  3  years  college 
3  or  »ore  years  college 


4393  (772) 

1327  (232) 

5720  (1002) 


1483  (922) 

124  (  82) 

1607  (1002) 


MEN  WOMEN 

212  232 
442  392 
352  382 


222  222 

402  392 

382  392 


About  22  percent  of  the  total  applicants  were  women.  However,  as  Table  14 
shows,  the  distribution  of  education  level  for  men  and  women  is  approximately 
the  same.  It  also  shows  that  approximately  78  percent  of  both  the  men  and 
women  have  some  education  beyond  high  school. 
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With  respect  to  aviation-related  experience,  23  percent  of  the  men  as  com¬ 
pared  to  only  8  percent  of  the  women,  identified  experience  which  could 
result  in  granting  extra  credit  points  for  selection  eligibility.  When 
education  and  experience  are  combined,  the  distribution  of  education 
level  remains  relatively  consistent  for  both  men  and  women  and  for  those 
with  or  without  aviation-related  experience. 

From  this  information  it  is  evident  that  the  level  of  education  for  men 
and  women  is  essentially  the  same  and  does  not  serve  to  differentiate 
between  the  applicants  based  on  their  sex.  The  distribution  of  aviation- 
related  experience  between  men  and  women  does  differentiate  between  the 
applicants  with  23  percent  of  the  men  potentially  being  eligible  for 
extra  credit  in  contrast  to  only  8  percent  of  the  women. 


Next,  the  ATC  applicant  sample  was  analyzed  in  terms  of  passing  or  failing 
the  CSC  test  in  relation  to  sex,  education  and  experience.  Table  15 
summarized  the  results  of  this  analysis. 


TABLE  15 


ATC  Applicant  Sample 


TOTAL 

CSC  TEST 

STATUS 

APPLICANTS 

PASS 

FAIL 

Men 

5720  (78%) 

52% 

48% 

Women 

1607  (22%) 

41% 

59% 

Total 

7327  (100%) 

(50%)* 

(50%)* 

EDUCATION  LEVEL 

High  School  or  less 

1581  (22%) 

32% 

68% 

Less  than  3  years  college 

2973  (40% 

49% 

51% 

3  or  more  years  college 

2757  (38%) 

61% 

39% 

Total 

7311  (100%) 

(50%)* 

(50%)* 

EXPERIENCE  LEVEL 

No  aviation-related  experience 

5876  (80%) 

45% 

55% 

Aviation-related  experience 

1451  (20%) 

68% 

32% 

Total 

7327  (100%) 

(50%)* 

(50%)* 

EDUCATION  &  EXPERIENCE 


Aviation-related  experience 


High  achool  or  less 

308 

(21%) 

52% 

48% 

Less  than  3  years  college 

632 

(44%) 

67% 

33% 

3  or  more  years  college 

508 

(35%) 

79% 

21% 

Total 

1448 

(100%) 

(68%)* 

(32%)* 

No  Aviation  experience  with: 


High  achool  or  less 

1273 

(22%) 

27% 

73% 

Less  that  3  years  college 

2341 

(40%) 

44% 

56% 

3  or  more  years  college 

2222 

128*2 

57% 

43% 

Total 

5836 

(100%) 

(45%)* 

(55%)* 

•Paaa/Fall  rates  for  the  Total  Croup 
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In  total,  about  50  percent  of  the  applicants  passed  the  CSC  test.  However 
proportionally  more  men  (52  percent)  passed  than  women  (Ml  percent). 


When  specific  education  levels  are  examined,  it  is  evident  that 
those  applicants  with  the  most  education  have  a  significantly  better 
chance  of  passing  the  test.  Applicants  with  3  or  more  jears  of  college 
passed  at  about  twice  the  rate  of  those  who  did  not  go  beyond  high  school. 
The  same  relationship  holds  also  for  experience.  Applicants  with  aviation- 
related  experience  also  pass  the  CSC  test  at  about  twice  the  rate  of  those 
without  experience. 


When  pass  or  fail  rates  are  examined  with  education  and  experience  combined, 
it  is  evident  that:  (1)  level  of  education  is  essentially  proportionally 
distributed  between  the  experienced  and  non-experienced  groups  (Table  14); 
and  (2)  experience  increased  the  pass  rate  in  each  of  the  education  groups. 
Applicants  with  no  college  or  experience  pass  at  a  27  percent  rate.  Those 
with' 3  or  more  years  of  college  and  aviation-related  experience  pass  at  a 
79  percent  rate. 


TABLE 

EXPERIENCE  AND  LEVEL  OF  EDUCATION 


CSC  TEST  PASS  RATE 
NO  WITH 


EDUCATION  LEVEL 

EXPERIENCE 

EXPERIENCE 

%  INCREASE 

High  School  or  less 

27% 

52% 

+25% 

Less  than  3  years  college 

44% 

67% 

+23% 

3  or  more  years  college 

57% 

79% 

+22% 

Two-way  analyses  of  yariance  were  carried  out  for  each  CSC  and  experimental 
test.  For  this  purpose  the  ".experience  group" was  further  classified  into 
subgroups  since  the  CSC  Rating  Guide  provides  different  levels  of  extra 
credit  based  on  specific  types  of  experience.  The  "key"  to  the  experience 
and  education  subgroups  are  as  follows: 

Aviation-Related  Experience 

0-1  No  aviation-related  experience 

2- 2  Comunication  experience  only 

3- 3  Non  pi7ot  with  IFR  or  Air  Defense  Command  (ADC)  experience 

4- 4  Non  pilot  with  VFR  ATC  experience 

5- 5  Pilot  with  IFR  or  ADC  experience 

6- 6  Pilot  experience  only 

7- 7  Pilot  with  VFR  ATC  experience 
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Education 


0-1  High  school  or  less 

2- 2  Less  than  3  years  of  college 

3- 3  3  or  more  years  of  college 

Table  17  provides  the  results  of  the  analysis  variance  for  the  CSC  total 
test  score.'  The  lowest  mean  score  (173.10)  is  for  the  "No  Experience  and 
No  College  "  Group;  the  highest  mean  score  (241.00)  is  for  the  "Pilot  with 
VFR  Experience  and  3  or  more  years  of  College"  Group.  The  analysis  of 
variance  shows  that  experience  (row)  and  education  (column) each  is  statis¬ 
tically  significant  (p^.01)  though  there  is  no  interaction  between  experi¬ 
ence  and  education. 

Further  analysis,  however,  shows  they  are  not  statistically  important  for 
the  total  group  since  only  1  percent  of  the  variance  in  CSC  total  score  was 
accounted  for  by  either  variable. 

TABLE  17 

ATC  TESTS  VS.  EDUCATION  AND  EXPERIENCE  GROUPS 
TOTAL  CSC  TEST  SCORE  (CSS  TOTAL  WT.) 
n  matrix  of  cases  with  non-mission  data  on  all  3  variables 


ED  GROUP 

GOES 

DOWN 

EXT. 

GR0LT 

GOES 

ACROSS 

O-i 

2-2 

3-3 

4-4 

5-5 

6-6 

7-7 

TOTAL 

N 

O-l 

-  1239 

34 

140 

28 

19 

67 

54 

156'. 

2-2 

-  2266 

75 

216 

47 

56 

180 

133 

2c-:. 

3-3 

-  2179 

70 

62 

25 

31 

2  30 

160 

27  5" 

SUV 

5684 

179 

438 

100 

106 

477 

347 

7311 

Feans  of  variable!  esc  total  kt 


ED  CROUP  GOES  DOWN  EXP.  GROUP  GOES  ACROSf 


<h3 

2-2 

3-3 

4-4  5-5  6^J>  7_-7 

MEAN 

O-l  - 
2-2  - 
3-3  - 

173.10 
197.40 

209.11 

179.53 

204.79 

221.59 

210.89 

218.55 

224.98 

201.96  198.68  191.07  211.56 
233.23  220.61  221.52  225.19 
222.72  246.10  236.17  241.00 

179.48 

2C2  .  6  - 
214.-7 

MEAN 

196.59 

206.56 

216.94 

221.85  224.13  224.31  230.36 

202.16 

NUMMARY  OF  ANALYSIS  OF 

VARIANCE  1 

Source 

Row 

248889.27 

2 

124444.63  51.25  0.000 

Col 

152242.89 

6 

25373.82  10.45  0.000 

R.BY.C 

44320.17 

12 

3693.35  1.52  0.109 

WITHIN 

CELL 

17701966.50 

7290 

2428.25 

However,  when  the  CSC  total  score  was  categorized  by  "pass"  or  "fail" 
groups,  education  and  experience  together  accounted  for  nearly  10  percent 
of  the  variance  in  pass/fail  status;  so  taken  together,  the  effects  of 
education  and  experience  are  important.  This  same  relationship  applies 
to  the  two  experimental  tests  as  well. 
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These  data  show:  (1)  that  applicants  with  avi at  ion- related  cKon^no?  have  a  hichcr 
probability  of  passing  the  CSC  test  (68  percent  as  oerj  arod  tc  45  percent  for  those 
without  exjvrience);  and,  (2)  that  few  wemon  haw-  aviation- related  experience.  These 
two  findings  taken  together,  however,  are  not  sufficient  to  explain  the  lower  pass 
rate  for  women  (41  percent)  as  oempared  to  men  (52  percent).  Apply  in  c  the*  sc-  probabilities 
to  the  experience  and  no  experience  crouds,  it  v*>uld  be  expected  that  752  weren  would 
have  passed;  only  663  did  pass.  For  men,  2,879  men  would  be  expected  to  pass;  2,980 
actually  did  pass.  Consequently,  simply  control  line  for  the-  fact  that  iren  have  more- 
aviation- related  experi ence  does  not  explain  why  women  score  lower  on  most  of  the 
tests. 


Intercorrelation  o:  each  of  the  tests,  education  and  ox; *  rienoe  are  provided  :r.  Tatir  IF. 


TABLE  18 


ATC  TEST  CORRELATIONS  —  TOTAL  SAMPLE 


POSITION  LABEL 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

T 

T 

TF 

T1 

TI 

C 

C:c 

TS 

24. S 

51.  S 

40. S 

57.5 

35. S 

SC.l 

TOT  A 

TS 

TS 

10. T 

DH.T 

ED.L 

PEO 

ED .  (■ 

XT.C 

LEa 

CORE 

CORE 

CORE 

CORE 

CORE 

OTAL 

L.VT 

10.  C 

10.  A 

OTAL 

DK.A 

DH.B 

G7AL 

EVEL 

AGO 

ROLE 

ROVE 

Cl  X 

T24  SCORE 
T52  SCORE 
T540  SCORE 
T157  SCORE. 
T135  SURF 
CSC  TOTAL 

esc  total  wt  . 

T510  C 
T51G  A 
T5 10  TOT/L 
DH.A 
DH.B 
DHTOTAL 
ED.  LEVEL 
ED.  LEVEL 
PEV  AW. 

ED. GROVE 
EXP.  CROUP 
EXP.  GPX 


KEYS: 


(6)  CSC  Total  Is  the  correlation  of  the  unweighted  CSC  test  scores 

(7)  CSC  Total  WT  16  the  correlation  of  the  weighted  CSC  test  scores 

(8)  T510  C  Is  the  correlation  of  MCAT  "aircraft  conflict"  scores 

(9)  T510  A  is  the  correlation  of  MCAT  "aptitude"  scores 

(14)  Ed  Level  Is  the  correlation  of  individual  levels  of  education  (01-13)  See  ref  (14)  C-1 

(15)  PEQ.  ACX.  is  the  correlation  of  aviat ion- related  experience  scales  derived  empirically.  See  ref 
(12)  Chapter  4  for  the  methodology  for  developing  the  scales 

(16)  Ed  Group  is  the  correlation  of  the  three  education  groups;  (1)  high  school  or  less;  (2)  less 
than  3  years  of  college;  (3)  3  or  ®ore  years  of  college. 

(17)  Exp  Group  is  the  correlation  of  the  seven  aviation— related  experience  groups 

(18)  Exp  GP.  X  is  the  correlation  of  the  group  with  aviation-related  experience  (excludlngthe  no 

experience  applicants) 


These  correlation  values  were  used  bo  correct  for  restrictions  in  range  on  test 
scores  of  individuals  hired  as  ATC  trainees  who  attended  the  FAA  Academy  durina  1976, 
1977  and  1978. 

Several  multiple  regression  analyses  were  carried  out  using  CSC  test  ’‘pass’1  or  "fail" 
as  a  criterion  for  the  applicant  croup.  Table  19  shews  the  results  for  the  present  CSC 
test  battery,  the  ord*?r  in  which  the  individual  tests  entered  the  analysis  and  their 
contribution  to  the  R2  which  measures  the  amount  of  variance  accounted  for  by  each  test. 
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TABLE  19 


ATC  REGRESSION  ANALYSIS  TOTAL  SANPLE 

riNAl  SLMlARY  0»  REGRESSION  ON  DEPENDENT  VARIABLE  CSC.  TOT. DR 


RULUplE  r  o.iom 
■U1.T  p  SQUARED  0.4483 
STO  ERROR  OP  1ST.  0.2963 
CONSTANT  0.0037 


ANALYSIS  OP 

VARIANCE  OP 
REGRESSION  s. 
RESIDUAL  7406. 


SLAV  OP 
SOUARES 
1203. 100 

646.664 


■can  p  pro*. 

SOUARE  RATIO  LEVEL 

240.620  >743. TS3  0. 

0.066 


MUM 

■ULT 

MULT 

Change 

VARIABLE 

ft. 

RTano 

•  ETA . 

FINAL 

f  MHEN 

VAR 

II 

* 

IN  ISO 

ENTERED 

•AM 

16*0* 

RTANO. 

f 

ENTERED 

MOV 

fto 

C  SHOWS 

COEF¬ 

OP 

COEP- 

TO 

OR 

IN 

DELETED) 

FICIENT 

* 

PJC1ENT 

DELETE 

Of  LI TED 

1 

0.711 

0.505 

0.505 

T 157 . SCORE 

0.014« 

0.0005 

0.6047 

•94793 

7959.294 

2 

0.  773 

0.597 

0  092 

T540.  SCOlIC 

0.0096 

0.0004 

0. 2530 

999.450 

1995.531 

a 

0.  7*4 

0.930 

0.033 

Tftisconc 

0.0152 

O.OOOT 

0.2034 

919.01 1 

993.970 

4 

0*02 

0.944 

0-014 

T 135. SCORE 

0.0093 

0.0004 

0.1416 

307.233 

293.097 

ft 

0.909 

0.949 

o.ooe 

T34. SCORE 

0.0050 

O.QOOS 

0.0647 

131.331 

191.391 

simple 

PARTIAL 

COR. 

COR.  IN 

NITH 

PINAL 

OCR. 

STEP 

0.7109 

0.6203 

0.9919 

0.2*36 

0.9092 

0.2666 

0.9390 

0.  1460 

0.9991 

0  -270 

Table  20  provides  the  results  of  the  two  experimental  tests  in  predicting 
the  CSC  pass  or  fail  status.  The  difference  between  the  for  these  tests 
(45  percent)  compared  to  the  CSC  tests  (65  percent)  indicate  that  they 
differ  in  some  degree  from  the  CSC  tests  in  what  they  measure. 

In  the  case  of  the  experimental  tests,  test  510  A  (MCAT  Aptitude)  which 
measures  aptitudes  similar  to  the  present  CSC  test  entered  the  regression 
first  and  accounts  for  40  percent  of  the  total  variance. 


TABLE  20 


4TC  REGRESSION  ANALYSIS  TOTAL  BANRLE 

Mnal  SL6N«ART  OP  REGRESSION  ON  OERENOERT  variable 


multiple  r 

NUlT  R  SOUARED 
STO  ERROR  OP  EST. 
CONSTANT 


0.04TS 
0 . A4S6 
O  S734 

0.66*6 


ANALYSIS  OP 

VARIANCE  OP 
REGRESSION  6. 
RESIDUAL  TAOB. 

SUB  OP 
SQUARES 
026714 
1027.(46 

■CAN 

SOUARE 

671.366 

0.166 

p 

RATIO 

1004.071 

PRO*. 

UVEL 

0. 

B  NUN 
T  VAR 
■  NON 
*  IN 

■ULT 

R 

■ULT 

• 

SO. 

CHANGE 

IN  660 

VA6IA6LE 

ENTERED 

1*  SMONS 
DtLCTtOl 

6. 

BA* 

COEP- 

PICIENT 

RTANO 

ERROR 

OP 

0 

SETA , 
RTANO. 

COEP- 
Pi Cl  ENT 

T  1 

a  > 

a  t 

0.061 

0.642 

0.444 

0.6*6 

0*36 

0.444 

0.666 

0.040 

0.007 

TRIO. A 
OH.tOTAL 

T610. e 

0.0326 

0.00*7 

0.6*41 

0.0016 

0.0006 

D.OOTO 

0.6766 

01670 

D.TTTO 

PINAL  » 

P  ENTERED 
TO  OR 

delete  deleted 

*16. 144  4616. 646 
444.647  B3R.S06 

06.164  61.164 


binrle  partial 

COR.  COR  IN 
■ITN  PINAL 
DEP-  STEP 

D.B6T*  0.6*76 
D.HIB  D.B6T* 
4.661*  O.TTO* 
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STUDY  OF  NEW  APPOINTEES  TO  THE  ATC  OCCUPATION,  1976-1978 


Objective.  This  study  was  initiated  by  the  FAA  Civil  Aeromedical  Institute 
(CAMI)  in  July  1976  to  evaluate  the  present  CSC  Test  Battery  and 
experimental  predictors  in  relation  to  the  success  of  newly  hired  ATC 
trainees  in  passing  Initial  Qualification  training  for  the  En  Route 
and  Terminal  ATC  options  at  the  FAA  Academy.  Upon  entry  on  duty,  new  ATC 
trainees  are  given  a  2-week  general  orientation  at  their  ATC  facility  (or 
regional  headquarters).  They  then  enter  Initial  Qualification  training  at 
the  FAA  Academy  in  Oklahoma  City  for  the  ATC  option  to  which  assigned.  On 
their  first  day  at  the  Academy,  they  are  given  a  series  of  experimental  tests 
and  questionnaires.  Their  participation  in  the  test  program  is  voluntary. 

The  experience  has  been  that  most  all  of  the  students  volunteered. 

Extensive  efforts  were  made  to  also  obtain  the  individuals'  test 
scores  on  the  five-part  CSC  battery  and  their  total  earned  ratings  which 
include  additional  credit  for  veterans  preference  and  aviaticn-related 
experience  as  provided  by  the  CSC  Rating  Guide.  This  information  was 
then  correlated  with  their  subsequent  training  status  to  evaluate  the 
CSC  and  experimental  tests  which  were  most  predictive  of  training 
success. 

Sample  Description.  During  the  period  July  1976  through  April  1973, 

3,008  students  entered  Terminal  and  En  route  ATC  training  at  the  FAA 
Academy.  Selected  characteristics  of  this  total  group  are  provided  in 

Table  2 1 . 


1  Ail  1.1  J  1 

SELECTION  CHARACTER  1  SI  lCS-HEAl.  CKlUT 


PASS  FAIL  WITHDRAW 


TOTAL  N 

N 

(%) 

N 

(%) 

N 

(%) 

Men 

2580 

2034 

(79%) 

436 

(17%) 

110 

(4%) 

Women 

428 

272 

(64%) 

128 

(30%) 

28 

(6%) 

Total 

3008 

2306 

(77%) 

564 

(19%) 

138 

(4%) 

Mo  Exp. 

834 

579 

(69%) 

206 

(24%) 

58 

(7%) 

Aviation-Related  Exp. 

1936 

1579 

(82%) 

296 

(15%) 

61 

(3%) 

Unknown 

229 

148 

(65%) 

62 

(27%) 

19 

(8%) 

Total 

3008 

2306 

564 

138 
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In  order  to  evaluate  the  CSC  and  experimental  predictors,  it  was  necessary 
to  have  complete  information  on  the  students.  The  1 38  students  who  with¬ 
drew  did  not  have  laboratory  criterion  scores  and  were  eliminated  from  the 
sample.  Similarly,  the  students  for  whom  prior  experience  information 
was  missing  were  also  rxrimid.  An  additional  824  students  were  excluded 
because  they  did  not  have  one  or  more  test  scores,  either  from  the  CSC 
test  battery  taken  prior  to  their  employment  with  FAA  or  were  students  who 
did  not  volunteer  to  take  the  experimental  tests. 


The  final  sample  used  for  the  analysis  after  these  exclusions  consisted  of 
1,827  students  distributed  as  shown  in  Table  22. 


tabu: 


SE 

LECTION 

CHARACTER! 

ISI  JCS-A.NALY'I  1  CAL 

S  AMI’LL 

PASS 

FAIL 

%  OF 

ORIGINAL 

TOTAL 

N  N 

(%) 

N 

(%) 

POPULATION 

Men 

1587 

1314 

(83%) 

273 

(17%) 

61.5% 

Women 

240 

162 

(68%) 

78 

(32%) 

56.1% 

Total 

1827 

1476 

781%) 

351 

(19%) 

60.7% 

No  Exp. 

558 

409 

(73%) 

149 

(27%) 

66.2% 

Aviation-Related  Exp. 

1269 

1827 

1067 

1476 

(84%) 

202 

351 

(16%) 

66.5% 

The  failure  rate  for  this  sample  (19.2  percent)  was  essentially  the  same 
as  for  the  total  population  (19.7  percent)  after  withdrawals  were  excluded. 

Predictors.  The  tests  used  in  this  study  were: 


-  Civil  Service  Test  Battery 

(CSC  24;  CSC  51;  CSC  135;  CSC  157;  and,  CSC  540) 

-  Directional  Headings  Test  (DHT)  Total  Score 

DHT  Part  A 
DHT  Part  B 

-  Multiplex  Controller  Aptitude  Test  (MCAT)  Total  Score 
MCAT  Aptitude  (A) 

MCAT  Conflicts^ ) 


Dial  Reading  (DL  RD) 

ATC  Occupational  Knowledge  Test  (OKT) 


The  Directional  Heading  Test  was  the  same  version  used  in  the  1977  study 
conducted  by  EPA  and  which  was  administered  to  the  7,500  ATC  appli¬ 
cants  in  1976-1977. 


The  MCAT  consisted  of  several  different  versions  (606  A;  606  B;  706  A; 

706  B;  607;  707)  which  varied  in  the  number  and  mix  of  questions 
(aptitude'/conflict)  and  in  the  length  of  time  allowed  ‘‘or  the  test. 

The  OtCT  was  form  1 0 IB  which  was  essentially  the  same  form  utilized  by  EPA 
in  their  1977  study.  The  Dial  Reading  Test  was  also  the  same  test  used  by 
EPA. 

Table  23  provides  the  means  and  standard  deviations  for  the  sample  of 
1,827  ATC  trainees  at  the  FAA  Academy  on  the  various  predictors  and  compares 
them  to  results  obtained  for  the  1976-1977  ATC  applicants  group  as  well  as 
with  the  ATC  sample  group  included  in  the  1977  ATC  Selection  Study  conducted 
by  EPA  where  comparable  statistics  were  obtained. 


IAJ5LL  2  3 


Cltt'I’AKAT  1 V 

i:  DliSCR  I 

PT1VL  Si' A 

T  1  SI  US 

1976-77  ATC 

APPLICANTS 

1977 

1976-78 

ATC 

CSC 

TOTAL 

CSC 

PASS 

EPA 

STUDY 

TRAINEES 

(N« 7412-6821) 

(N»3690-3340> 

(N-1323-1229) 

(N-1827) 

TEST 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

MEAN 

SD 

CSC- 24 

39.66 

9.6 

44.80 

6.8 

NA 

47.07 

6.7 

CSC- 51 

26.65 

6.7 

30.74 

3.8 

NA 

31.90 

3.3 

CSC- 540 

28. 98 

13.1 

37.70 

10.1 

NA 

42.88 

9.7 

CSC-157 

29.29 

10.3 

36.65 

6.0 

NA 

38.17 

6.4 

CSC- 13 5 

21.50 

8.6 

27.00 

5.1 

NA 

29.31 

4.2 

MCAT  (A) 

16.59 

5.7 

20.27 

4.1 

NC 

23.14 

4.5 

MCAT  (C) 

9.17 

4.2 

11.38 

3.9 

NC 

15.74 

4.1 

MCAT  TOT 

22.76 

9.1 

31.65 

7.2 

NC 

38.87 

7.6 

DHT  A 

24.20 

12.0 

30.69 

9.8 

NA 

32.68 

9.0 

DHT  B 

22.80 

11.7 

29.25 

8.9 

31.9 

6.9 

31.63 

9.0 

DHT  TOT 

47.00 

22.6 

59.93 

17.3 

NA 

64.34 

17.0 

DL-RD 

NA 

NA 

39.7 

9.4 

40.97 

9.2 

OKT 

NA 

NA 

74.9 

13.6 

66.96 

16.3 
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(Table  23  Continued) 


NA  -  Statistics  not  available  since  the  tests  were  either  not 

administered  to  these  groups  or  could  not  be  obtained  from 
available  records.  In  the  case  of  DHT,  for  the  1  977  Kl’A 
Study  only  Part  B  was  scored  and  used  in  the  analysis. 

NC  -  Statistics  not  comparable.  In  the  1977  F.PA  Study,  two  forms 
of  MCAT  were  administered  to  the  test  sample  (706  A,  60b  A). 
In  the  analysis  all  "aptitude"  questions  from  both  forms  were 
combined  for  a  total  aptitude  score  (mean  =  42.8:  SD-6.2)  as 
were  all  "conflict"  questions  (mean  =  29.4:  SD-5.6). 


Criterion  Measure.  In  this  analysis,  the  ATC  laboratory  average  score  was 
used  as  the  criterion  measure  of  ATC  training  success.  Selection  oi  this 
criterion  was  based  on  several  considerations.  First,  prior  studies 
demonstrated  that  these  scores  were  the  most  reliable  predictors  of  subse¬ 
quent  success  as  an  Air  Traffic  Controller.  Second,  the  laboratory  trainin'.' 
phase  consists  of  a  series  of  operational  air  traffic  control  problems  in 
which  students  must  demonstrate  their  ability  to  apply  the  academic  know¬ 
ledge  and  skills  acquired  in  training.  Finally,  this  phase  is  conducted  on 
a  pass/fail  basis.  The  laboratory  score  constitutes  63  percent  of  the 
students  total  score  in  the  laboratory  phase  of  training.  Almost  all  of 
of  the  students  who  fail  Initial  Qualifications  training,  do  so  in  the 
laboratory  training  phase.  It  should  be  pointed  out  that  during  the  period 
from  July  1976  through  April  1978  several  changes  were  made  in  the  laboratory 
training  phase  which  affected  the  pass/fail  ratio  of  students.  In  July  l‘i76 
the  weight  given  to  the  laboratory  average  was  increased  from  35  percent  to  63 
percent.  In  September  1976  the  number  of  graded  lab  problems  was  increased 
from  four  to  six;  and  in  May  1977  "procedural  errors"  were  incorporated  in 
the  grading  for  failure  to  handle  aircraft  in  a  timelv  manner  during,  the 
problem  exercise.  In  order  to  accomodate  these  changes  and  the  differences 
between  the  Enroute  and  Terminal  laboratory  procedures,  the  laboratory 
averages  were  converted  to  standard  scores  and  data  from  the  two  ATC  options 
combined.  The  standardized  laboratory  average  criterion  is  labeled  "FI. Ah." 

The  reliability  of  the  laboratory  average  for  Enroute  training,  is  .80 
(2,223  students)  and  for  the  Terminal  option  ,81  (1,982  students). 

A  detailed  discussion  of  the  ATC  Initial  Qualifications  training  program 
and  follow-on  training  at  the  ATC  facility  level  for  the  Terminal  and 
Enroute  options  are  provided  in  references  23,  24,  and  25. 


44 


Analytical  Methodology.  The  first  step  in  the  analysis  was  the  calculation 
of  descriptive  statistics.  The  remaining  analyses  presented  several  unique 
problems. 

Several  different  experimental  forms  of  the  MCAT  were  employed  in  the 
testing  at  the  FAA  Academy  and  the  order  of  administration  varied  for 
each  form.  Consequently ,  it  was  necessary  to  standardize  the  scores  for 
these  tests.  Since  MCAT  706  A  was  also  administered  to  6,821  ATC  appli¬ 
cants  by  0PM  in  1976-1977,  the  scores  on  the  various  forms  of  MCAT 
used  by  CAMI  were  standardized  by  linear  conversions  using  the  same  metric 
as  MCAT  706  A.  The  order  effect  was  handled  by  using  the  scores  from  MCAT 
706  A  given  first.  References  19  and  20  provide  a  more  detailed  dis¬ 
cussion  on  the  methodology  used  for  these  conversions. 


The  second  problem  involved  in  evaluating  the  CSC  and  experimental  tests 
was  the  restriction  in  range  effect* 21)  since  criterion  (ZLAB)  information 
was  available  only  on  those  individuals  who  were  hired.  This  results  in  a 
spurious  low  correlation  of  CSC  test  scores  with  the  criterion.  In  order 
to  adjust  the  restricted  correlations  so  they  would  reflect  the  relationship 
between  the  tests  and  the  criterion  for  the  applicant  group,  the  correla¬ 
tions  were  corrected  for  their  restriction  in  range.  Usual  methods  for 
this  correction  in  the  three  variables  case  assume  that  unrestricted  infor¬ 
mation  is  available  only  on  the  variable  used  for  selection  or  the  third 
incidental  variable  but  not  both.  With  the  information  derived  from  the 
1976-1977  ATC  applicant  population,  unrestricted  data  were  available  on 
both  variables.  A  modified  procedure  was  developed  to  make  use  of  all  of 
the  available  information  in  correcting  for  range  restriction.  A  full 
discussion  of  the  procedure  and  the  derivation  of  equations  is  provided  in 
reference  22. 

The  unrestricted  correlation  matrix  from  the  total  1976-1977  ATC  applicant 
group  used  for  correcting  for  range  restriction  is  provided  in  Table  24. 

The  means,  standard  deviations  and  sample  sizes  (N's)  were  provided  in 
Table  23  except  for  the  Earned  Rating. 


TablelA.  Unrestricted  Correlation  Matrix  Fro*  the  CSC 


12  3  4  5  9  10  11  12  13  !<•  19 

T  T  T  T  f  T  T  T  EAR 

VARIABLE  24  .  51.  540.  157.  135  .  510  51  0  510.  OH.  NED. 

SCORE  SCORE  SCORE  SCORE  SCORE  .A  .C  TOTAL  OH. A  DH.B  TOTAL  RATE 


1 

r 24. Score 

HX1 

33 

56 

50 

2 

f 51. Score 

33' 

"mo 

46 

59 

3 

T540. Score 

56 

46- 

'"■MO 

59 

4 

T157.  Score 

50 

59 

59"' 

— -iQO 

5 

T135.  Score 

51 

54 

58 

63" 

9 

T510.A 

52 

60 

63 

63 

10 

T510.C 

44 

42 

56 

49 

11 

T510. TOTAL 

53 

57 

65 

62 

12 

OH. A 

46 

54 

55 

53 

13 

0H.B 

47 

54 

56 

56 

14 

OH. TOTAL 

49 

57 

59 

57 

19 

CAMEO  RATE 

66 

75 

78 

85 

51 

52 

44 

53 

46 

47 

49 

66 

54 

60 

42 

57 

54 

54 

57 

75 

58 

63 

: 

65 

55 

56 

59 

78 

63 

63 

49 

62 

53 

56 

57 

85 

66 

52 

65 

54 

56 

59 

77 

6*" 

52 

■400 

69 

94 

65 

67 

69 

77 

69" 

"K>a 

89 

54 

55 

57 

60 

65 

94 

89" 

"M0 

66 

67 

70 

76 

54 

65 

54 

66" 

"MO 

82 

95 

66 

58 

67 

55 

67 

82" 

"MO 

95 

69 

59 

69 

57 

70 

95 

95" 

"MO 

72 

77 

77 

60 

76 

68 

69 

72" 

1M0 
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The  Earned  Rating  for  the  applicant  group  was  derived  by  taking  the  raw 
weighted  CSC  total  score,  transmuting  it  to  a  metric  scale  of  0-100  and, 
where  appropriate,  adding  veterans  preference  (5  points)  and  additional 
credit  for  experience  (0,  5,  10,  or  15  points  based  on  the  CSC  Rating 
Guide  criteria)  to  the  scores  of  applicants  who  passed  the  CSC  Test 
Battery.  This  Earned  Rating  is  the  basis  on  which  0PM  ranks  and 
places  eligible  candidates  on  a  register  from  which  selections  are 
made  in  accordance  with  0PM  regulations. 

Table  25  provides  the  restricted  correlation  matrix  for  the  1 , 827  ATC 
trainees  at  the  ATC  Academy.  The  means  and  standard  deviations  for  this 
sample  were  previously  provided  in  Table  23*  It  should  be  note  that  the  stand¬ 
ardized  (Z)  scores  were  used  for  the  MCAT  tests  in  deriving  the  intercor¬ 
relation  (mean  =  0;  SD  =  +  1). 

TABLE  25 

RESTRICTED  CORRELATION  MATRIX  USED  IN  REGRESSION  ANALYSES 


-  -  -  l  SCORES  - 


CSC  24 

CSC  540 

CSC  157 

DHT  A 

DHT  B 

DHT  TOT 

MCAT  A 

MCAT  C 

MCAT  TOT 

ZLAB 

OL  ‘RD 

CSC 

24 

1.000 

0.333 

0.140 

0.245 

0.255 

0.271 

0.214 

0.146 

0.204 

0.097 

0.343 

CSC 

540 

0.333 

1.000 

0.145 

0.199 

0.222 

0.227 

0.204 

0.190 

0.221 

0.096 

0.289 

CSC 

157 

0.140 

0.145 

1.000 

0.090 

0.140 

0.126 

0.241 

0.164 

0.229 

0.071 

0.261 

DHT 

A 

0.245 

0.199 

0.090 

1.000 

0.803 

0.950 

0.258 

0.268 

0.296 

0.207 

0.338 

DHT 

B 

0.255 

0.222 

0.140 

0.803 

1.000 

0.949 

0.284 

0.268 

0.308 

0.223 

0.341 

DHT 

TOT 

0.271 

0.227 

0.126 

0.950 

0.949 

1.000 

0.282 

0.282 

0.316 

0.227 

0.356 

Z  MCAT  A 

0.214 

0.204 

0.241 

0.258 

0.284 

0.282 

1.000 

0.596 

0.903 

0.246 

0.481 

Z  MCAT  B 

0. 146 

0.190 

0.164 

0.268 

0.268 

0.282 

0.596 

1.000 

0.882 

0.250 

0.445 

Z  MCAT  TOT  0.204 

0.221 

0.229 

0.296 

0.308 

0.316 

0.903 

0.882 

1.000 

0.277 

0.518 

£  LAB 

0.097 

0.096 

0.071 

0.207 

0.223 

0.227 

0.246 

0.250 

0.277 

1.000 

0.272 

DL- 

RD 

0.343 

0.289 

0.281 

0.338 

0.341 

0.356 

0.481 

0.445 

0.518 

0.272 

1.000 

The  correlations  of  particular  interest  are  those  between  the  tests  and  the 
"ZLAB"  criterion.  These  are  zero  order  validity  coefficients.  The  effects 
of  restriction  are  immediately  apparent  in  the  low  correlations  between  the 
CSC  tests  used  in  the  actual  selection  and  the  ZLAB  criterion.  The  two 
highest  zero  order  coefficients  are  MCAT  TOTAL  (.277)  and  DL-RD  .272, 
neither  of  which  was  restricted  directly  by  selection  of  the  trainees. 

Two  CSC  tests,  51 — Spatial  Patterns  and  135 — Following  Oral  Directions 
were  omitted  from  the  matrix.  They  were  eliminated  based  on  the  negative 
skew  previously  discussed.  Extreme  selection,  such  as  in  the  case  of  ATC 
trainees,  results  in  a  sharp  reduction  in  the  variance  in  the  selected 
groups  as  was  shown  in  Tables  13  and  23  which  compares  the  means  and 
standard  deviations  for  the  total  ATC  applicant  group  and  the  ATC  appli¬ 
cant  Pass  Group,  and  the  ATC  trainee  groups. 
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When  correcting  for  restriction  in  range,  the  difference  between  the  applicant 
group  variance  and  the  selected  group  variance  is  enployed  as  a  measure  of  the 
amount  of  curtailment  that  occurs  due  to  selection.  It  was  not  determined  if  the 
skew  resulted  in  a  violation  of  the  linearity  assurption ;  however,  the  extreme 
disparity  between  the  two  variances  for  CSC  51  and  CSC  135  resulted  in  a  corrected 
correlation  that  was  much  higher  than  the  other  corrected  correlations .  In  this 
analysis,  if  the  correlations  for  CSC  51  and  CSC  135  were  used  in  the  multiple 
regression  analysis,  none  of  the  other  tests  either  independently  or  in  combination 
added  anything  significant  to  the  multiple  R  beyond  CSC  51  and  CSC  135.  These  results 
were  considered  spurious  and  CSC  51  and  CSC  135  were  excluded  from  the  following 
multiple  regression  analyses. 


Table  26  provides  the  estimated  unrestricted  correlations  (as  well  as  the  actual 
unrestricted  correlation  from  the  CSC  sairple) .  The  correlations  of  primary  interest 
are  the  correlations  of  the  tests  with  the  ZLAB  criterion.  After  correcting  for 
range  restriction,  the  MCAT  at  .531,  Dial  Reading  at  .466,  and  Directional  Heading 
at  .461  have  the  highest  zero  order  validity  coefficients. 


Table  26  Unrestricted  and  Corrected  Correlations  Used  in  Regression  Analyses 


CSC  24 

CSC  5ft0 

CSC  157- 

0HT  1 

DHT  2 

CSC  24 

troacL 

.560 

.500 

.4 60 

.470 

CSC  54° 

.560 

~~~T>oaa 

.590 

.550 

.560 

CSC  157 

.500 

.590 

"-*>000 

.530 

.560 

DHT 

,ft60 

.550 

.530 

"*-r0Qa_ 

.820 

DHT  2 

.ft70 

.560 

.  560 

.820 

'TrOOC. 

DHT  TOT 

.ft90 

.590 

.570 

.950 

.950 

MCAT  A 

.520 

.630 

.630 

.650 

.670 

MCAT  C 

.ftftO 

.560 

.490 

.540 

.550 

MCAT  TOT 

.530 

.650 

.620 

.660 

.670 

ZLAB 

.342 

.386 

.399 

.432 

.446 

DL*RD 

.515 

.501 

.502 

.498 

.510 

T  TOT 

MCAT  A 

MCAT  C 

MCAT  TOT 

2LAE 

DL-RD 

.490 

.520 

.440 

.530 

.342 

.515 

.590 

.630 

.560 

.650 

.386 

.501 

.570 

.630 

.490 

.620 

.399 

.502 

.950 

.650 

.540 

.660 

.432 

.498 

.950 

.670 

.550 

.670 

.446 

.510 

TreeCL__ 

.690 

.570 

.700 

.461 

.522 

.690 

Treca. 

.690 

.940 

.503 

.  61 1 

.570 

.690 

.890 

.475 

.551 

.700 

.940 

.890 

HOCK) 

.531 

.635 

.461 

.503 

.475 

.  531 

.466 

.466 

.522 

.611 

.551 

.635 

Treoa 

Results.  The  next  step  in  the  analyses  was  to  employ  the  unrestricted  and  corrected 
correlations  in  a  series  of  step-wise  multiple  regression  procedures.  Each  model 
was  a  refinement  of  the  previous  model,  with  test  scores  regressed  on  ZLAB.  The  first 
model  excluded  total  scores  for  DHT  and  I^CAT,  since  these  are  the  sum  of  their  past 
scores  that  were  included.  Results  are  presented  in  Table  27. 


Tabla  27 

INITIAL  REGRESSION  MODEL 


t  -  0.5689  BSQ  .  0.3236 


V 

BETA 

B 

CSC  24 

0.0071 

0.0011 

CSC  340 

-0.0066 

-0.0007 

CSC  157 

0.0555 

0.0043 

DHT  A 

0.0513 

0-.0057 

DHT  B 

0.0912 

0.0101 

MCAT  A 

0.1452 

0.0322 

MCAT  C 

0.1668 

0.0407 

DL  'ID 

0.1856 

0.0201 

■EC.  00NST. 

• 

•3.0582 

r-IEST  1 
■SO  TULL 
BSQ  BEDCCED 
DimnjKz 

Dm  •  7. 


TOTAL  MODEL  WITH  PAST  SCORES 
0.3236  Modal  1 

0.0000  Modal  0 


0.3236 
DTD  •  1800. 
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D-RATI0  •  123.020 


K  0.0001 


In  succeeding  models,  CSC  540  was  eliminated  because  of  its  lack  of 
contribution,  and  total  scores  were  used  for  DHT  and  MCAT  instead  of  part 
scores.  The  resulting  multiple  "R"  was  .5672.  Then  Dial  Reading  was 
eliminated  because  of  its  marginal  value  in  the  1972  and  1977  FAA  ATC 
Specialist  studies,  and  DHT  was  eliminated  because,  in  its  current  format 
(very  brief  and  highly  speeded)  it  is  not  suitable  for  operational  use. 
Results  are  presented  in  Table  28.  The  initial  model  yielded  a  multiple 
"R"  of  .5689,  the  final  model  yielded  a  multiple  "R"  of  .5407. 


Table  28 

FINAL  REGRESSION  MODEL 
R  -  0.5407  RSQ  -  0.2924 

V  BETA  B 


CSC  24 

0.0608 

0.0090 

CSC  157 

0.0964 

0.0074 

MCAT  T 

0.4391 

0.0570 

REG.  CONST. 

m 

03.2045 

F-TEST 

24,  157,  MCAT 

TOTAL 

RSQ  FULL 

0.2924 

Model 

5 

RSQ  REDUCED  - 

0.0000 

Model 

0 

DIFFERENCE  - 

0.2924 

DFN  -  2. 

DFD  -  1800. 

F- 

RATIO  -  371.890 

P  *  0.0001 

A  factor  analysis  (principal  axis  analysis  with  varimax  rotation)  was 
performed  to  explore  the  characteristics  of  the  the  test  scores.  The 
results  are  provided  in  Table  29. 


Table  29  Factor  Analyeie  on  Teet  Score*  (Principal  Axia  Analysi*  -  Varimax  Rotation) 


Variable* 

Factor  1 

Factor  2 

Factor  3 

Factor  4 

Factor  5 

Factor  6 

CSC  24 

0.9731 

-0.0550 

-0.0261 

-0.0181 

0.2583 

-0.0211 

CSC  5*0 

0.4396 

0. 3568 

-0.1129 

-0.5921 

0. 5849 

-0.0564 

CSC  157 

0. 3578 

0.0785 

-0.7302 

0.0079 

0.5883 

-0.0350 

DHT  T 

0.2638 

-0.1068 

0.0238 

-0.0347 

0. 7255 

0.0543 

MCAT  T 

0.2715 

-0. 1042 

-0.0066 

-0.0305 

0.7357 

0.0522 

ZLAB 

0.2834 

-0. 1036 

0.0143 

-0.0415 

0.7666 

0.0563 

DL-RD 

0.3137 

-0.0885 

-0.0788 

-0.0704 

0.  7  740 

0.0315 

Z  variance  accounted  for 

22.7188 

2.5335 

7.9083 

5.1392 

42.9871 

0.2097 

Two  rather  clear  structures  underlying  the  data  with  the  orthogonal 
rotation  appear.  Factor  1  and  Factor  5  account  for  22.72  and  42.99  percent 
of  the  variance  respectively.  Factor  5  contains  the  largest  loading  for 
all  tests  and  ZLAB  with  the  exception  of  CSC  24.  It  is  also  notable  that 
the  division  seems  to  occur  on  both  factors  between  the  CSC  tests  (24,540, 
157)  and  the  experimental  tests  (MCAT,  DHT,  DR-RD)  and  ZLAB. 
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The  beta  weiqhts  derived  from  the  Final  Regression  Model  (Table  28) 
were  converted  to  raw  score  weights  and  then  assigned  unit  weights.  The 
following  equation  constitutes  the  composite  score: 


Yt  =  1(CSC  24)  +  2  (CSC  157)  +  4 (MCAT) 

where  Yc  =  the  composite  test  score  for  the  battery.  Using  the  unit  weights, 
the  multiple  R  derived  was  .5354  compared  to  .5407  with  beta  weights.  The 
R2  value  derived  was  .2867  with  unit  weights  compared  to  .2924  using  beta 
weights. 


A  cross-validation  study  was  performed  to  investigate  the  stability  of 
the  results.  The  sample  was  randomly  separated  into  two  groups,  and  the 
weights  derived  from  the  first  sample  were  applied  to  the  second  sample  to 
observe  shrinkage  in  the  multiple  R.  Distribution  statistics  and  intercor¬ 
relations  for  the  two  groups  are  presented  in  Table  30. 


TABLE  30  DISTRIBUTION  BY  RACE  AND  SEX  AND 
DESCRIPTIVE  STATISTICS 


SAMPLE  1 


SAMPLE  2 


Men 

Women 

Total 

Men 

Women 

Total 

black 

47 

17 

64 

BUCK 

HISPANIC 

14 

3 

17 

45 

16 

61 

AM.  INDIAN 

0 

1 

1 

HISPANIC 

15 

4 

19 

ORIENTAL 

6 

1 

7 

AM.  INDIAN 

0 

0 

0 

ESKIMO 

1 

0 

1 

ORIENTAL 

7 

1 

8 

OTHER 

730 

94 

824 

ESKIMO 

1 

1 

2 

OTHER 

723 

101 

824 

TOTAL 

798 

116 

914 

TOTAL 

791 

123 

914 

MEAN 

S.D. 

MEAN 

S.D. 

ZLAB 

0.028 

1.007 

ZLAB 

-0.020 

0.990 

CSC  24 

46.998 

6.871 

CSC  24 

47.026 

6.853 

CSC  157 

38.490 

6.538 

CSC  157 

38.252 

6.244 

MCAT 

35.608 

7.451 

MCAT 

35.686 

7.307 

CORREUTIONS 

CORRELATIONS 

ZLAB 

1.000  0.328 

0.402 

.0.537 

ZLAB 

1.000  0.326 

0.396 

0.527 

CSC  24 

1.000 

0.500 

0.530 

CSC  24 

1.000 

0.500 

0.530 

CSC  157 

1.000 

0.620 

CSC  157 

1.000 

0.620 

MCAT 

1.000 

MCAT 

1.000 
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The  regression  equation  for  the  first  sample  is  presented  in  Table  31. 


Table  31 


CROSSVALIDATlOfi  CSC  SELECTION  STUDY 


R  -  0.5450 

RSQ  - 

0.2970 

VARIABLE 

BETA 

B 

CSC  24 

CSC  157 

MCAT 

0.0354 

0.1023 

0.4546 

0.0052 

0.0156 

0.06i5 

REC.  CONST. 

- 

3.0113 

F-TEST 

RSQ  FULL 

RSQ  REDUCED 
DIFFERENCE 
DFN  -  2. 

CROSSVALIDATION 
«  0.2675  Model  1 

-  0.0000  Model  0 

-  0.2970 

DFT  *  913.  F-RATIO 

-  192.847 

PCO.OOOl 

Unit  weights,  derived  from  the  beta  values  for  Sample  1  were  computed. 
These  unit  weights  (CSC  24  *  1;  CSC  15?  *  2;  and,  MCAT  *  4)  were  then  used 
to  compute  the  multiple  Rs  and  K^s  in  Sample  1  and  Sample  2  with  the 
following  results: 


TABLE  32  CALCULATED  MULTIPLE  R  VALUES 
Sample  Multiple  R  R2 

1  .5381  .2895 

2  .5292  .2801 

Full  discussion  of  the  analysis  and  results  on  the  CSC  and  experimental 
tests  for  appointees  to  the  ATC  occupation  in  1976-1978  is  provide  in 
reference  20  .  The  results  from  these  analyses  supported  the  conclusion 
that  CSC  24,  CSC  157  and  MCAT  tests  be  used  in  combination  as  a  selection 
battery  for  screening  applicant  air  traffic  controllers.  Some  information 
was  provided  that  indicated  a  modified  form  of  the  Directional  Headings 
test  suitable  for  operational  use  and  a  Dial  Reading  Test  should  be  explored 
in  future  studies. 

In  the  analyses  of  tests  in  relation  to  ATC  trainees  hired  during 
1976-1978  discussed  thus  far,  the  ATC  Occupational  Knowledge  Test  (OKT)  was 
excluded  from  the  various  regression  analyses.  The  OKT  is  a  "job-knowledge 
specific"  test  and  is  not  intended  for  use  as  a  "pass"  or  "fail"  exam  in 
determining  applicant  eligibility  for  employment  consideration.  Rather  its 
use  would  be  limited  to  measuring  ATC  related  experience  and  knowledge  as 
a  basis  for  granting  extra  point  credit  to  those  applicants  who  pass  the 
competitive  test  battery.  Consequently,  the  ATC  Occupational  Knowledge 
Test  (OKT)  was  analyzed  as  a  parallel  effort. 


* 
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Objective.  The  analysis  was  conducted  to  determine  the  relationship  of  OKT  to: 

the  present  method  of  granting  extra  credit  based  solely  on 
ATC  related  experience. 

.  Pass/fail  status  during  Initial  ATC  Qualification  training. 

.  An  ATC  selection  test  battery  comprised  of  CSC  tests  24,  157 

and  MCAT'and  the  combined  estimated  validity  of  the  test  battery 
and  OKT. 

Sample  Description.  The  sample  for  the  analysis  of  OKT  was  the  same  group  of 
1827  ATC  trainees  (Table  22)  that  was  used  for  regression  analyses  of  the  CSC 
and  experimental  tests  previously  discussed. 

Predictor .  The  OKT  (form  101B)  is  a  100-item  multiple  choice  test  covering 
air  traffic  control  regulations,  communication,  flight  service  station  work, 
navigation  aids,  weather,  and  radar.  The  score  is  the  total  number  of  correct 
responses.  The  mean  and  standard  deviation  for  the  1827  ATC  trainees  was  66.96 
and  16.3  respectively  (Table  23).  The  Kuder-Richardson  -  2C  internal  cor.sis- 
tancy  estimate  of  reliability  yielded  a  reliability  cofficient  of  .95  for  a 
sample  of  ATC  new  hires. 

Criterion.  The  same  criterion  measurejATC  laboratory  average  score  (ZLAB) , 
that  was  used  in  the  multiple  regression  study  was  used  in  analyses  of  OKT. 

Analytical  Methodology.  The  1827  ATC  trainees  were  grouped  into  various 
AIC  related  experience  categories  based  on  their  response  to  a  biographical 
questionnaire  administered  on  the  first  day  of  training.  Four  types  of 
ATC  related  experience  were  identified. 

1.  Other  experience  (included  air  defense  command,  communications 
operator  and  prior  ATC  training  without  ATC  operational  experience) . 

2.  Pilot  experience 

3.  VFR  or  non-radar  ATC  experience 

4.  IFR  or  radar  control  ATC  experience 

These  four  types  of  experience  (together  with  "No  ATC  related  experience") 
were  then  used  to  classify  each  of  the  1827  trainees  into  16  experience  sub¬ 
groups  which  provided  for  combinations  of  experience  as  listed  below. 


592 


1 

No  ATC  related  experience 

558 

2 

Other  experience 

34 

3 

Pilot 

297 

4 

Pilot  +  Other 

92 

5 

VFR  only 

99 

6 

VFR  +  Other 

27 

7 

VFR  +  Pilot 

28 

8 

VFR  t  Pilot  +  Other 

37 

9 

IFR  only 

86 

10 

IFR  +  Other 

29 

11 

IFR  +  Pilot 

18 

12 

IFR  +  Pilot  +  Other 

11. 

13 

IFR  +  VFR 

275 

14 

IFR  +  VFR  +  Other 

95 

15 

IFR  +  VFR  +  Pilot 

57 

16 

Total 

IFR  +  VFR  +  Pilot  +  Other 

84 

1827 

389 

191 


144 


511 


Because  of  the  small  number  of  trainees  in  some  of  the  subgroups,  they 
were  combined  into  the  following  five  major  groups: 


Group  1 
Group  2 
Group  3 
Group  4 
Group  5 


No  experience  +  Other  experience  only  (592) 
Pilot  or  Pilot  +  Other  experience  (389) 
VFR  +  any  additional  experience  except  IFR  (191) 
IFR  +  any  additional  experience  except  VFR  (144) 
IFR  +  VFR  +  any  additional  experience  (511) 


The  test  scores  for  OKT  were  correlated  with  experience  and  later  correlated 
with  the  criterion  ZLAB.  Statistical  analyses  of  each  of  the  five  experience 
groups  in  relation  to  OKT  score  ranges  and  pass/fail  status  was  also  conducted. 

Results.  The  correlation  of  OKT  scores  with  experience  was  .61.  OKT  correlated 
with  the  ZLAB  criterion  at  .22  versus  .11  for  experience  credit.  This  indicates 
that  while  OKT  is  closely  related  to  experience,  it  is  more  predictive  of  success 
than  the  present  method  of  giving  credit  based  on  experience  in  accordance  with 
the  OPM  Rating  Guide 


The  results  of  the  statistical  analysis  of  eacl  of  the  five  experience 
groups,  the:jr  OKT  scores  and  ATC  training  pass/fail  status  is  provided  in 
Table  33. 


Table  3? 


Failure  Rates  by  OFT  .>core  Ranges  and  OkT  -leans  by  txperlence  Groups 


Score 

Ot  her 

Total  i  Fall 

Pilot 

Total  1  Fail 

Vh  n 

Total  l  Fail 

IFR 

Total  1  Fall 

VFH 

Total 

♦  IFK 
&  Fail 

T0T»L 

Total  1  Fall 

0-59 

4  30 

29  3* 

o7 

2o.9'l 

32.01 

1  1 

27  .3* 

23 

43. 81 

556 

29  71 

60-64 

37 

27 .0t 

47 

23  41 

M 

42  11 

9 

33  3* 

23 

34 . 81 

135 

29  61 

65-69 

44 

25  OS  I 

55 

9  IS 

r  1 

23  ol 

13 

23  It 

46 

23  91 

179 

20  11 

70-74 

34 

20  61 

77 

HO; 

)0 . 4  J 

13 

1o.7J 

0  \ 

16.41 

226 

16  4V 

75-79 

22 

67 

10. 4; 

i' 

1  1.51 

34 

11  .81 

1 17 

14.51 

277 

12.61 

80* 

59  2 

12.0s 

10.51 

6  S 

13-1 

Ji2_ 

6.11 

2'.J 

7.  It 

454 

3.41 

TOTAL 

2o.91 

339 

15.21 

hi 

21.51 

144 

13.21 

511 

14.34 

1827 

1921 

Means 

on 

OKT  51  49  699** 

std .  dew.  14.8  10.9 


7?n 

114 


76.28 

9.5 


73.0  66.96 

9-7  16. 3 


While  the  mean  OKT  score  for  the  total  1827  ATC  trainees  is  66.96 
(SD  m  16. 3),  examination  of  Table  33  shows  marked  differences  for  the  five 
experience  groups  and,  within  each  group  by  OKT  score  range  as  shown  below. 


Experience  Group 

N 

Mean 

SD 

Percent  of  Failure 

No  +  Other  Experience 

592 

51.49 

14.8 

26.9% 

VFR  +  Any  other  (except  IFR) 

191 

72.25 

11.4 

21.5% 

Pilot  +  Other  Experience 

389 

69.94 

10.9 

15.2% 

VFR  &  IFR  +  Any  other 

511 

78.00 

9.7 

14.3% 

IFR  +  Any  other  (except  VFR) 

144 

76.28 

9.5 

13.2% 

Of  the  total  1827  ATC  trainees,  359  (19.6%)  were  in  ATC  related  experience 
groups  which  presently  earn  extra  credit  under  the  CSC  Rating  Guide  for  this 
experience  and  who  scored  less  than  70  on  the  OKT.  The  failure  rate  for  the 
359  was  26.2%  as  compared  to  28.2%  for  the  511  trainees  with  no  (or  other) 

ATC  related  experience  who  also  scored  less  than  70  on  OKT. 

As  can  be  seen  from  Table  33,  as  OKT  score  ranges  get  higher,  the  pro¬ 
portion  of  failures  generally  continues  to  drop  in  each  of  the  five  experience 
groups.  Rather  than  using  the  present  method  for  granting  extra  credit  for 
experience,  the  use  of  OKT  scores  would  be  significantly  more  predictive  of 
ATC  training  success. 

Thus  far,  OKT  data  reported  has  compared  the  OKT  test  scores  to  the  present 
method  of  crediting  ATC  related  experience  and  each  of  these  in  terms  of  their 
prediction  of  ATC  training  pass/fail  status.  For  OKT  to  be  truely  useful,  it 
must  contribute  to  predicting  ATC  training  success  over  and  above  what  can  be 
achieved  by  a  new  selection  test  battery. 
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As  previously  shown,  a  test  battery  comprised  of  CSC  24,  CSC  157  and 
MCAT  had  a  multiple  R  of  .54  which  accounted  for  29%  of  the  variance  on 
ZLAB.  If  the  correlations  of  OKT  with  ZLAB  and  the  other  test  battery 
scores  are  corrected  for  restriction  in  range  using  Thorndike's  formula  7, 
then  the  multiple  regression  coefficient  including  OKT  can  be  estimated. 

The  new  Multiple  R,  calculated  including  OKT  as  well  as  the  three  tests 
previously  included  in  the  battery  would  be  .60  and  36%  of  the  ZLAB  variance 
would  be  accounted  for.  This  increase  is  significant  (F  =  204.3  p  4.001), 
indicating  that  the  addition  of  extra  points  using  OKT  scores  would  signi¬ 
ficantly  improve  prediction  and  that  OKT  should  be  used  as  a  basis  for 
determining  extra  credit.  The  analysis  of  OKT  summarized  here  is  discussed 
in  more  depth  in  reference 


DEVELOPMENT  OF  THE  MULTIPLEX  CONTROLLER  APTITUDE  TEST 


EXPERIMENTAL  TEST  FORMS .  Development  of  the  Multiplex  Controller  Aptitude 
Test  was  based,  in  part,  on  previous  research  efforts  directed  to  developing 
performance  indices  for  air  traffic  controllers  during  1970-1971  (2b).  From 
this  research,  the  initial  versions  of  the  Controller  Decision  Evaluation 
(CODE)  film  tests  were  developed  and  used  in  the  1972  study  of  Air  Traffic 
Control  Specialists  conducted  by  Education  and  Public  Affairs  (12). 

The  CODE  test  used  a  motion  picture  film  to  present  simulated  air  traffic 
in  real  time  as  they  cross  a  controller's  display  scope,  together  with  a 
mileage  scale,  lines  to  represent  airways  and  intersections,  and  a  data  table 
to  provide  information  on  each  aircraft's  identity,  route,  speed  and  altitude. 
The  test  was  administered  in  a  free  response  mode,  with  the  task  being  to 
predict  violations  of  aircraft  separation  standards  (conflicts)  as  early  and 
accurately  as  possible. 

Experimental  administration  of  three  motion  picture  films  were  carried  out. 
Analysis  revealed  that  most  of  what  was  measured  in  each  45  minute  film  could 
be  derived  from  only  7  or  8  of  the  pairs  of  conflicting  aircraft.  In  addi¬ 
tion,  there  was  considerable  idle  time  during  testing  which  offered  the  oppor¬ 
tunity  to  measure  more  than  simply  conflict  performance.  Consequently,  test 
development  procedures  were  initiated  to  measure  not  only  this  skill  but  others 
as  well. 

An  initial  modification  was  to  develop  a  structured  test,  using  a  slide  pro¬ 
jector  to  place  conflict-type  questions  on  the  screen.  However,  when  items 
were  assembled  into  a  film/slide  version,  half  of  the  test  time  remained  idle 
and  this  provided  the  opportunity  for  test  subjects  to  change  their  answers  as 
the  test  progressed  and  the  correct  answers  became  obvious.  It  was  found  that 
by  presenting  a  new  item  every  45  seconds  the  examinees  were  kept  sufficiently 
busy  so  that  this  situation  could  be  controlled. 

Ample  information  is  available  on  the  scope  and  table  to  ask  a  variety  of 
questions.  Items  were  written  that  utilized  the  available  detailed  information 
to  measure  such  aptitudes  as  direction  following,  table  reading,  spatial  visual¬ 
ization,  and  arithmetic  reasoning  regarding  separation  distances  and  times. 

Some  items  were  very  simple.  Others  were  written  in  a  multi-factor  format  to 
increase  their  difficulty.  For  example,  initial  instructional  questions  on  how 
to  read  the  table  were  very  easy,  but  a  complex  problem  such  as  estimating 
travel  time  in  minutes  between  two  aircraft  (horizontal  separation)  required 
awareness  of  distances  across  the  scope,  reading  the  table  to  determine  their 
speed,  and  mathematical  computation  to  determine  their  rate  of  closure. 

Items  were  administered  experimentally,  the  relation  between  item  types  and 
total  test  homogeniety  determined,  and  this  ratio  used  to  determine  the  number 
of  items  per  type  to  include  in  the  test.  A  result,  for  example,  was  inclusion 
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of  twice  as  many  time-distance  separation  items  as  compass  heading  items  in 
the  test.  Conflict  item  locations  had  been  established  by  mean  journeyman 
response  times  when  targets  were  at  certain  locations.  Aptitude  items  were 
placed  in  the  remaining  positions,  45  seconds  apart,  alternating  from  one 
type  to  another  and  spiraling  to  increasing  levels  of  difficulty  as  testing 
progressed.  About  40  percent  of  the  items  were  conflicts,  and  the  other  60 
percent  were  aptitudes.  As  films  4,  6  and  7  each  provided  a  different  pattern 
of  air  traffic,  three  different  forms  of  the  test,  MCAT  4,  6  and  7  were 
prepared. 

Versions  of  the  CODE  test,  the  film/slide  version  of  MCAT,  and  other  selec¬ 
tion  measures  were  administered  to  students  of  the  Navy  Air  Traffic  School, 
Memphis,  and  grades  of  these  students  obtained  as  they  progressed  through 
training.  Analyses  included  intercorrelations  and  rotated  factor  loadings 
of  the  various  tests  with  each  other  and  with  such  school  grades  as  laboratory 
flight  plans,  control  tower,  and  radar  control  problems,  as  these  grades  pro¬ 
vide  the  greatest  range  in  criterion  scores  for  predicting  FAA  student  scores. 
The  film/slide  structured  version  had  correlations  with  criteria  that  were 
as  high  or  higher  than  those  of  the  free  response  CODE.  These  results  en¬ 
couraged  further  development  of  MCAT. 

A  next  step  in  test  development  was  to  create  a  "slides  only"  version,  in 
essence  taking  pictures  of  the  screen  each  time  a  new  question  appeared. 

Pacing  of  the  slide  presentation  was  the  same  as  in  the  film  plus  slide 
version,  so  the  amount  of  target  movement,  how  far  an  aircraft  moved  from 
question  to  question,  was  unchanged.  Free  response,  film/slide  and  all-slide 
versions  were  administered  to  students  at  the  Air  Force  Controller  School, 
Keesler,  and  grades  of  the  students  obtained  as  they  progressed  through  training 
The  all-slide  version  generally  showed  higher  correlations  with  the  criteria 
than  either  the  free  response  or  film/slide  versions. 

During  these  test  administrations,  it  became  evident  that  the  use  of  motion 
picture  or  slide  projector  equipment  would  not  be  feasible  for  operational 
test  administration.  The  effect  of  equipment  failure,  room  size,  lighting, 
seating  arrangement,  and  other  factors  could  not  be  controlled  at  the  many 
locations  where  the  Office  of  Personnel  Management  administers  the  test  to 
applicants. 

A  next  step  was  to  print  the  slide  versions  in  paper-pencil  format.  Three 
forms  were  prepared  and  administered  to  several  populations  including: 
entering  ATC  students  at  the  FAA  Academy  during  all  of  1976;  ATC  Specialists 
participating  in  the  1977  study  conducted  by  Education  and  Public  Affairs, 

Inc.,  under  FAA  contract;  and,  7000  ATC  applicants  tested  by  the  CSC  in  tha 
fall  of  1976  and  spring  of  1977. 

Each  form  was  then  lengthened  to  55  items  and  administered  to  all  students 
at  the  FAA  Academy  from  1977  until  June  1978;  students  at  various  colleges 
and  universities;  and,  students  at  the  Army  Air  Traffic  Controller  School, 

Ft.  Rucker.  All  three  forms  were  administered  to  each  of  the  Army  students 
and  differences  in  mean  performances  among  forms  were  found  that  could  effect 
form  comparability.  Also,  student  performances  improved  considerably  on  the 
second  form  they  were  administered,  indicating  that  learning  was  still  taking 
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place,  but  there  was  very  little  increase  in  performance  on  the  third  form. 
Differences  in  learning  rate  among  students  was  noted,  and  this  ability  was 
recognized  to  be  another  potentially  useful  measure. 

Development  of  various  forms  of  MCAT  has  resulted  in  reliability  correlations 
that  vary  with  the  restriction  in  range  of  the  scores  for  the  populations 
tested.  Correlations  between  early  forms  were  .31  to  .50  for  ATC  Academy 
students  who  had  greater  restrictions  in  range  of  their  scores;  .60  to  .  *-> 
for  USAF  ATC  students;  and,  .87  to  .90  for  non-controller  FAA  employees  whc 
more  nearly  approximate  the  pjpulation  for  whom  the  test  was  designed.  Vali¬ 
dity  of  the  MCAT  test  with  the  different  criteria  used  for  prediction  or 
"ATC  success"  have  been  identified  in  the  various  studies  previously  discussed. 

A  more  detailed  description  of  the  development  of  experimental  forms  of  the 
MCAT  test  is  found  elsewhere  (14). 

The  test  is  designed  to  measure  the  skills  of  the  air  traffic  controller  appli¬ 
cant  within  an  ATC  simulated  setting.  What  is  accomplished,  in  effect,  is  to 
teach  persons  a  set  of  simplified  ATC  rules,  and  then  test  how  well  they  can 
apply  those  rules  in  specific  situations.  Test  subjects  spend  eight  minutes 
studying  the  directions,  and  then  start  to  answer  the  questions.  Because  the 
air  traffic  controller  activity  is  complicated  and  rather  difficult  to  learn, 
the  test  is  designed  so  that  initial  items  are  very  easy,  and  then  progress  in 
an  order  of  increasing  difficulty.  The  time  limits  are  deliberately  established 
to  make  the  test  a  speeded  test.  While  usual  aptitude  battery  design  practice 
is  to  cluster  items  into  homogeneous  subgroups,  in  this  test  the  content  of  the 
items  is  alternated  from  one  type  to  another  and  they  spiral  to  increasing 
levels  of  difficulty.  This  mode  of  presentation  is  found  in  only  a  few  tests 
such  as  the  Stanford-Binet.  A  result  shown  statistically  is  that  the  non* 
confliction  items  show  unexpectedly  high  commonality  with  those  items  requiring 
detection  of  impending  conflicts.  Apparently  the  detection  of  conflicts 
requires  the  same  skills  and  operations  measured  by  the  different  kinds  of 
aptitude  items  presented. 

Figure  3  provides  the  basic  format  of  the  MCAT.  About  40  percent  of  the  total 
55  questions  ask  for  identification  of  potential  conflicts,  or  violations  of 
separation  standards.  The  remaining  questions  ask  for  such  information  as 
differences  in  the  routes  of  pairs  of  aircraft,  how  far  apart  two  aircraft  may 
be  at  a  given  moment,  their  compass  headings,  and  if  they  will  come  into  conflict. 

DEVELOPMENT  OF  PARALLEL  MCAT  TEST  FORMS.  For  operational  purposes,  multiple 
forms  of  a  test  are  necessary  for  retest  purposes  and  control  over  compromise. 
Based  on  previous  item  analysis  and  experimental  test  form  development,  each  of 
the  three  experimental  test  forms  (407A,  607A  and  707A)  were  split  into  two 
parts  by  placing  the  odd  items  in  one  part  and  the  even  items  into  a  second 
part.  These  individual  parts  were  then  placed  in  various  combinations  to  provide 
six  available  forms.  Each  form  was  a  55  item  test  and  separated  into  two  parts. 
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FIGURE  3 


THE  MULTIPLEX  CONTROLLER  APTITUDE  TEST 
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A  typical  example  of  the  kind  of  information  provided  in  the  test  problems 
is  given  above.  The  drawing  shows  the  particular  flight  paths  that  aircraft  must 
follow.  Changes  in  routes  can  occur  only  at  the  intersection  between  two  routes. 
Each  x  depicted  on  the  routes  represents  an  aircraft  traveling  in  a  particular 
direction  indicated  by  the  trailing  dots. 


To  the  side,  a  table  containing  critical  flight  information  about  each  air¬ 
craft  is  provided.  The  number  next  to  each  x  on  the  drawing  allows  the  matching 
of  the  flight  information  to  the  correct  aircraft.  The  flight  information  lists 
the  altitude,  speed  in  miles  per  hour,  and  route  that  each  aircraft  is  flying. 
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Each  part  required  a  person  to  learn  its  unique  mix  and  pattern  of  traffic  and 
make  predictions  as  to  traffic  behavior  and  potential  conf lictions .  Part  1 
was  administered  allowing  20  minutes  for  completion;  15  minutes  was  allowed 
for  Part  2.  The  six  forms  were  then  administered  to  two  sample  populations; 
new  ATC  hires  entering  the  FAA  Academy  for  training  after  June  20,  1973, 
and  to  approximately  7800  applicants  for  ATC  work  who  took  the  present  ATC 
test  administered  by  OPM  in  October-November  1978.  Table  14  provides  the 
descriptive  statistics  for  each  group  for  each  form  of  the  MCAT  test. 

I'AllI-K  34 
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The  test  is  for  use  with  OPM  ATCS  applicants,  and  those  are  the  data  of  primary 
interest.  The  FAA  Academy  students  data  are  included  for  future  test  develop¬ 
ment  work,  to  estimate  applicant  scores  on  new  forms  of  the  test  when  experimen¬ 
tal  data  are  gathered  on  FAA  Academy  students.  While  all  forms  are  quite  com¬ 
parable  for  the  ATC  applicant  sample,  item  revisions  will  be  made  in  forms 
6o7e  and  7e4o  to  increase  their  comparability. 

Many  different  scoring  methods  have  been  explored  during  development  of  the  test. 
The  test  could  be  scored  for  Rights  and  Wrongs,  with  separate  scores  for  Conflicts 
and  each  of  the  Aptitudes,  and  with  differential  combining  weights  for  all  these 
subscores.  The  test  could  be  scored  for  the  rate  of  improvement  from  its  early 
to  later  parts.  Various  multiple  regression  studies  indicate  that  little  is 
gained  from  maintaining  separate  scores  for  Conflicts  and  Aptitudes  as  was  done 
in  the  1977  study  conducted  by  Education  and  Public  Affairs,  Inc.  Consequently, 
performance  on  all  items  have  been  combined  into  a  single  score  in  subsequent 
analyses.  The  data  also  showed  that  little  was  gained  through  use  of  a  combina¬ 
tion  of  "Rights  and  Wrongs"  instead  of  "Rights"  only. 

For  a  group  of  617  ATC  students  at  the  FAA  Academy  tested  between  June  and 
December,  1978,  a  total  "Rights"  score  test-retest  correlation  of  .60 
between  comparable  two  part  35  minute  forms  of  MCAT  was  obtained.  A  planned 
doubling  of  the  test  length  providing  a  total  test  time  of  90  minutes  should 
increase  the  reliability  coefficient  to  .75  for  this  restricted  group  and 
provide  an  adequate  level  of  reliability  for  the  ATC  applicant  population. 
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The  following  summarizes  the  development  of  the  MCAT  test. 


CODE  -  film  versions  4,  6  and  7.  Scoring:  1971,  40  items; 

1974,  Stanine:  1975,  corrected  item  scores  -  best  items.  Un¬ 
structured  free  response. 

.  Film  Slide  MCAT  -  July  1975.  Time  controlled  for  answering  each 

item.  MCAT  406  FS  -  41  items,  606  FS  -  43  items,  706  FS  -  53  items. 
Structured  multiple  choice. 

.  All  slide  MCAT  -  October  1975.  Time  controlled  for  answering  each 
item.  MCAT  406  AS  -  41  items,  606  AS  -  43  items,  706  AS  -  53  items. 
Structured  multiple  choice. 

.  Paper  Pencil  MCAT  -  January  1976.  MCAT  406A  -  41  items,  606A  - 
43  items  (25  minutes  test  time)  706A  -  53  items  (30  minutes  test 
time).  Structured  multiple  choice. 

.  Paper  Pencil  MCAT  -  April  1976.  Time  controlled  for  answering  each 
five  minute  group  of  items.  Form  606B  -  43  items,  706B  -  53  items. 
Structured  multiple  choice. 

.  Paper  Pencil  MCAT  -  January  1977.  Time  controlled  for  working  total 
test  (35  minutes) .  407A,  607A,  707A  each  55  items.  Structured 

multiple  choice. 

.  Paper  Pencil  MCAT  -  June  1978.  Parallel  test  forms.  Two  part  tests 
with  each  part  presenting  a  unique  pattern  of  air  traffic.  Part  1  - 
20  minutes,  Part  2-15  minutes.  Forms:  4o6e;  4e6o;  6e7o;  6o7e> 
7e4o;  7o4e ;  each  test  55  items.  Structured  multiple  choice. 
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DEVELOPMENT  OF  THE  AIR  TRAFFIC  CONTROLLER  OCCUPATIONAL  KNOWLEDGE  TEST 


The  ATC  Occupational  Knowledge  Test  (OKT)  is  a  subject  element,  paper-pencil, 
machine  scorable,  multiple  choice  type  test.  It  measures  important  aspects 
of  knowledge  associated  with  prior  experience  that  are  related  to  successful 
performance  as  an  air  traffic  controller  specialist.  The  subject  matter 
areas  included  are  Air  Traffic  Control  Regulations,  Terminal  Air  Traffic 
Control,  Enroute  Air  Traffic  Control,  Communications,  Flight  Service  Stations, 
Navigational  Aids,  Weather,  and  Radar. 

Experimental  Test  Development 

The  initial  ATC  Occupational  Knowledge  Test  was  designed  to  measure  the 
acceptability  of  an  applicant's  claimed  experience  for  qualifying  at  the 
GS-9  level  or  above.  There  was  such  a  wide  variety  among  applicant's  claims 
of  experience  that  they  were  difficult  to  evaluate,  and  the  OKT  would  provide 
scores  to  verify  the  acceptability  of  claimed  experience.  The  first  step  in 
test  development  was  a  two-week  workshop  in  June  1970,  in  which  eighteen 
journeyman  air  traffic  control  specialists  from.  Terminals,  Centers  and 
Flight  Service  Stations  participated.  The  workshop  was  to  teach  these 
people  the  specifications  for  the  test  and  help  them  write  items  under  the 
training  and  supervision  of  representatives  of  FAA  Headquarters.  This 
committee  examined  the  exact  types  of  qualifying  experience  that  were  given 
credit,  and  determined  the  specifications  for  the  test  to  permit  its  applica¬ 
tion  to  verify  claimed  experience.  Then  over  300  items  were  written,  and 
reviewed  and  edited  by  the  Examination  and  Certification  Section  of  the  ATC 
Academy,  Oklahoma  City.  Two  test  booklets  were  created.  Form  15  containing 
150  of  the  items,  and  Form  16  containing  160  items. 

These  forms  were  administered  to  two  classes  of  students  on  the  day  they 
reported  for  training  at  the  ATC  Academy,  and  to  samples  of  GS-7  and  GS-9 
Air  Traffic  Control  Specialists  in  Flight  Service  Stations,  Terminals,  and 
Centers.  The  tests  were  found  to  correlate  highly  with  past  experience 
patterns,  had  acceptable  validity  coefficients  with  training  success  for 
Center,  Terminal,  and  Flight  Service  Station  training,  and  demonstrated 
ability  to  differentiate  Air  Traffic  Controller  from  Flight  Service  Station 
job  incumbents.  The  results  of  the  experimental  testing  were  analyzed  at  the 
individual  item  level,  and  these  data  used  for  creating  multiple  forms  of 
the  test. 

Since  planned  initial  application  was  a  test  for  use  as  a  partial  basis  for 
determining  applicants'  qualification  for  Terminal  and  Center  job  entrar. 
above  the  GS-7  level,  a  first  step  was  to  eliminate  items  from  considers  a 
that  had  been  found  lacking  by  experts  in  the  specialty  area.  The  performance 
of  virtuoso  air  traffic  controllers  was  compared  to  performance  of  entering 
ATC  Academy  students  on  an  item  by  item  basis.  Prime  criteria  for  am 
item's  selection  were  (1)  its  ability  to  differentiate  between  the  two  groups 
and  (2)  the  item's  difficulty  level.  The  easiest  100  of  the  most  discriminating 
items  were  selected  for  inclusion  in  Form  101  -  Experimental  amd  the  next 
most  difficult  100  items  were  assembled  as  Form  201. 
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A  more  detailed  discussion  of  the  initial  development  of  the  Occupational 
Knowledge  Test  is  found  elsewhere  (30).  Subsequent  to  1970,  FAA  discon¬ 
tinued  hiring  ATC  applicants  at  grades  above  GS-7  and  work  on  developing 
the  ATC  Occupational  Knowledge  (OKT)  was  deferred  until  1975,  when  further 
research  was  initiated  to  improve  ATC  selection. 

In  August,  1975,  a  battery  of  experimental  selection  tests,  including  Forms 
101  and  201  of  the  ATC  Occupational  Knowledge  Test,  were  administered  to 
109  students  of  the  Navy  Air  Traffic  School,  Memphis.  Student  scores  on 
Form  101  were  distributed  against  their  week  in  training  when  tested,  and 
these  showed  increases  in  test  performance  levels  as  the  students  gained 
experience.  Grades  of  these  students  were  obtained  as  they  progressed 
through  training.  Analyses  included  intercorrelations  and  rotated  factor 
loadings  of  the  various  tests  with  each  other  and  with  school  grades  such 
as  academic  and  laboratory  grades  and  course  average.  Distribution  statistics 
plus  correlations  with  scores  on  selected  variables  are  presented  in  Table  35. 
Mean  scores  on  Form  201  are  ten  points  lower  than  on  Form  101,  and  the  Form 
201  total  score  correlations  with  other  measures  are  similarly  depressed. 

TABLE  35 

PERFORMANCE  RELATIONS  BETWEEN  OKT  AND  SELECTED  VARIABLES 


Navy  ATC  Students 
August  1975 


OKT  101 


OKT  201 


Mean 

SD 

Mean 

SD 

TOTAL  GROUP  (N-109) 

61.6 

8.9 

51.5 

6.7 

CORRELATION  WITH: 

£ 

r 

Reading 

CM 

O 

1 

-.01 

Education  Level 

.27 

.10 

MCAT 

.42 

.22 

OKT  101 

— 

.16 

OKT  201 

.16 

— 

School  Grades 

Academic  Average 

.48 

.25 

Block  II  Composite 

.  28 

.  16 

Block  III:  Basic  Lab 

.  28 

.  06 

Block  III :  Advanced  Lab 

.27 

.13 

Course  Average 

.43 

.25 

In  the  factor  analysis,  student  performance  on  the  ATC  Occupational  Knowledge 
Test  (OKT)  showed  little  or  no  communality  with  their  ability  to  read  and 
comprehend  the  material  and  very  little  relation  with  their  mechanical-spatial 
aptitudes.  OKT-101  had  high  loadings  on  the  Air  Traffic  Controller  Perfor¬ 
mance  factor,  the  Control  Tower  Operator  factor,  and  the  Laboratory  factor. 
0KT-201  had  a  similar  factor  structure  but  with  less  loading  on  each  of  the 
factors,  perhaps  as  a  function  of  restriction  in  variance,  since  it  was  a 
rather  difficult  test  for  this  population. 
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A  battery  of  experimental  tests  including  ATC  Occupational  Knowledge  Test 
101  Experimental  was  also  administered  to  forty-one  students  at  the  Air 
Force  Air  Traffic  Controller  School,  Keeslei  AFB,  during  October  1975. 
Distribution  statistics  and  correlations  with  selected  school  grades 
are  presented  in  Table  36.  The  performance  of  these  students  were  quite 
similar  to  those  of  the  Navy  Air  Traffic  Controller  School  sample. 

TABU  36 

"FRKOKMANCF  RELATIONS  BETWEEN  OKI  AN D_S  ELECT  El)  VARIABLES 

'  '•••'  ' -  OKI  101 

LSAF  AiC  STUDENTS  '  ”  Standard 

October  197 c  Mean  Deviation 

Total  Group  (N=41)  63.66  16.34 

School  Grades,  Final 

School  Correlations  L 

Block  I  Basic  FAA  Certification  .42 

Block  II  Basic  Operations  Specialist  .15 

Block  III  Control  Tower  Operator  .32 

Block  IV  Radar  Controller  (6CA/PAR)  .68 

Based  on  the  results  of  administering  Form  101  and  201  of  the  Occupational 
Knowledge  Test  to  these  groups,  as  well  as  to  controllers  at  selected  FAA 
terminal  and  enroute  facilities  in  December  1975,  the  best  100  items  were 
selected  to  create  Form  10 IB  of  the  OKT.  This  form  was  part  of  the  experi¬ 
mental  test  battery  administered  to  FAA  facility  personnel  and  ATC  Academy 
students  in  the  1977  selection  study  conducted  by  Education  and  Public 
Affairs  Inc  (.6).  Distribution  statistics  and  correlations  of  OKT  101 

Supervisory  Assessments  obtained  in  the  19-7  study  for  field  personnel 

are  given  in  Table  37.  Correlation  with  supervisory  assessments  for  field 
ATC  personnel  were  among  the  highest  found  for  variables  in  the  experimental 
test  battery. 


TABLE  37 

DISTRIBUTION  STATISTICS  AND  CORRELATIONS 

~~atc  occupational  knowledge  TEST  -  101 B 

(Facility  ATC  Specialists  -  100  items,  97  keyed) 


ATC  Option 

and  Year  Hired 

IFR  1969-70 
197  3-74 
TOTAL 

VFR  1969-70 
1973-74 
TOTAL 

ARTCC  1969-70 
1973-74 
TOTAL 

FSS  1969-70 
1973-74 
TOTAL 

ALL  OPTIONS  1969-70 
1973-74 
TOTAL 


Correlation  with 
Supervisory  Assessment 
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Comparable  data  for  a  group  of  new  ATC  trainees  entering  the  FAA  Academy  in 
1976  are  provided  in  Table  18. 


TABLE  38 

DISTRIBUTION  STATISTICS  AND  OORFELATItJC 
ATC  OCCUPATIONAL  KNOKLEDGE  TCST  -  101B 


(Acadery 

ATC  Trainees  -  100 

items , 

97  keyed) 

Academy  Classes  4-8 

Cbrrelaticn  with 

Z  Lab  Score 

N 

MEAN 

SD 

Jan  -  June  1976 

Tbtal  Terminal  and 

Enroute  by  Subqroips 

803 

67.3 

16.42 

Aviation  experience 

.40 

349 

76.6 

10.88 

No  aviation  experience 

.27 

452 

60.2 

16.23 

Man 

.31 

700 

68.7 

16.06 

Wtmen 

.11 

103 

57.5 

15.62 

Minority 

.36 

86 

67.5 

16.51 

Non-Minority 

.29 

710 

68.7 

16.06 

Terminal 

.31 

372 

70.9 

15.44 

ESiroute 

.32 

431 

64.2 

16.33 

As  would  be  expected,  mean  scores  on  OKT  for  this  trainee  group  were  lower 
and  the  standard  deviations  higher  than  for  the  more  experienced  facility 
ATC  specialists  hired  in  1969-70  or  1973-74.  Correlations  with  the  non¬ 
radar  laboratory  training  scores  (Z  scores)  for  the  Academy  were  highest 
for  those  with  prior  aviation  experience  and  for  minorities.  Correlations 
for  the  terminal  and  enroute  option  were  also  among  the  highest  found  for 
variables  in  the  experimental  test  battery. 

The  ATC  OKT  101B  was  also  evaluated  in  relation  to  present  methods  for 
granting  extra  credit  for  aviation  related  experience.  The  ATC  Occupational 
Knowledge  Test  101B  was  administered  to  784  ATC  trainees  who  entered  the 
FAA  Academy's  16-week  ATC  training  program  between  July  and  December,  1976. 
All  trainees  completed  a  pre-employment  questionnaire.  Based  on  responses 
to  the  questionnaire,  the  trainees  were  assigned  to  one  of  three  experience 
groups  in  accordance  with  evaluations  of  claimed  experience  as  made  by  the 
Office  of  Personnel  Management  rating  procedures.  It  was  found  that  while 
scores  on  the  OKT  were  highly  correlated  with  claimed  experience  (.64),  OKT 
had  a  higher  correlation  with  successful  completion  of  the  non-radar  lab 
(.25)  than  did  claimed  experience  (.12).  It  was  determined  that  use  of  an 
OKT  score  of  75  or  above  to  assign  extra  credit  would  result  in  a  failure 
rate  of  3.1  percent  for  those  receiving  credit,  while  use  of  the  present 
experience  rating  would  result  in  a  failure  rate  of  7.6  percent  for 
new  hires  who  now  receive  extra  credit  for  experience.  The  results  held 
up  for  a  cross-validation  sample  of  432  trainees  who  entered  the  Academy 
during  1977.  The  full  results  of  this  analysis  of  OKT  is  provided  elsewhere 


In  order  to  obtain  data  on  the  ATC  applicant  groups  within  available  test¬ 
ing  time,  a  60  item  form  of  OKT  (101-C)  was  developed  from  form  101B. 

This  was  administered  to  two  groups  of  ATC  applicants  during  September- 
November  1978,  together  with  parallel  forms  of  MCAT  after  they  had  completed 
the  existing  operational  ATC  test  battery  used  by  OPM.  One  group  consisted 
of  5331  scheduled  applicants.  The  second  group  was  669  "walk  in"  applicants 
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in  testing  sessions  arranged  to  encourage  women  and  minorities  to  apply  for 
ATC  work.  Mean  scores  for  the  scheduled  applicants  on  the  60-item  OKT  (101C) 
was  24.9  with  a  standard  deviation  of  11.80  and  for  the  "walk  in”  group,  20.1 
and  9.71  respectively.  OKT  Form  101C  was  also  administered  to  groups  of  ATC 
trainees  at  the  ATC  Academy  during  1978.  From  these  analyses  it  was  con¬ 
cluded  that  the  ATC  occupational  Knowledge  Test  Form  101B  has  high  reliability 
(.91)  and  that  the  slight  loss  of  reliability  with  reduction  in  test  length 
would  not  be  of  consequence. 

Development  of  Parallel  Test  Forms 

The  purpose  of  the  OKT  is  to  define  the  domain  of  ATC  Knowledge  that  is 
demonstrated  by  applicants  for  the  occupation  as  a  basis  for  granting  extra 
credit  to  those  who  successfully  pass  the  basic  ATC  selection  tests  admini¬ 
stered  by  OPM.  For  operational  usage,  multiple  forms  are  needed  to  meet 
retest  requirements  and  provide  some  capability  for  control  over  compromise. 

An  essential  step  in  the  development  of  alternate  forms  was  to  define  the 
types  of  items  and  number  of  items  of  each  type  to  include  m  each  succeeding 
form,  to  structure  the  domain  of  knowledge  to  make  it  consistent  with  the 
knowledge  requirements  of  air  traffic  control.  This  was  accomplished  pri¬ 
marily  by  determining  what  types  of  knowledge  were  retained  after  training 
and  early  apprenticeships  experience  and  were  known  by  nearly  all  full  per¬ 
formance  level  controllers.  Table  ht  identifies  the  steps  involved  in  deter¬ 
mining  the  final  domain  of  ATC  occupational  knowledge  included  in  OKT  and 
provide  a  basis  for  developing  future  comparable  forms. 


As  the  Federal  Aviation  Regulations  state  that,  to  become  certified  as  a 
Terminal  ATCS ,  the  person  must  pass  a  written  knowledge  test  on  (1)  Flight 
Rules  in  FAR  Part  91  (Air  Traffic  Control  Rules) ,  (2)  Terminal  Traffic 
Control  Procedures,  (3)  EnRoute  Traffic  Control  Procedures,  (4)  Communica¬ 
tions  Operating  Procedures,  (5)  Flight  Assistance  Service,  (6)  Air  Navigation 
and  Aids  to  Navigation,  and  (7)  Aviation  Weather,  the  FAA  Basic  Certification 
Test  was  designed  to  test  knowledge  in  each  of  those  subject  matter  areas. 

The  original  form  of  the  certification  test  is  listed  first  in  Table  38,  and 
the  table  shows  that  an  equal  percentage  of  items  (14  percent)  were  allocated 
to  each  subject  matter  area.  For  development  of  the  OKT,  radar  questions  were 
added  to  the  item  pool  (line  2) ,  and  the  best  items  selected  (lines  3-6)  to 
measure  candiates'  possession  of  the  knowledge  required.  The  percentage 
allocation  per  subject  matter  area  is  presented  in  line  7  for  version 
101B  of  the  OKT.  Each  of  the  seven  subject  matter  areas  is  further  defined 
by  subject  sub-elements  which  are  identified  in  Table  •<’.  Table  .1  shows 
the  classification  of  items  in  Form  101B  by  element  and  sub-element.  Numbers 
in  the  cells  are  item  number.  If  the  item  number  is  underlined,  that  item 
was  found  to  be  known  by  90  percent  or  more  of  the  1975  sample  of  50  full- 
performance  level  Air  Traffic  Controllers. 
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TAKU:  >9 

ATC  OCCUPATIONAL  KNOWLEDGE  TEST  -  SUBJECT  MATTER  ELEMENTS 
PERCENT  OF  ITEMS  -  EACH  ELEMENT 


Inflight 

Coesnuni- 

Flight 

Air 

Airport 

Traffic 

caticns 

Assistance 

Navigation 

Number 

ATC 

Traffic 

Control 

Operations 

Service 

&  Aids  tc 

Aviation 

of 

Form  of  OK  Test 

Pules 

Procedures 

Procedures 

Procedures 

Procedures 

Navigation 

Weather 

I  terns 

1.  Origin*!  FAA  Basic 

14% 

14% 

14% 

14% 

14% 

14% 

14% 

140 

Certification  Te*t 
All  Items 

2.  Fona  16 

12* 

12* 

25 

12* 

12* 

12* 

12* 

160 

All  Items 

3-  Fonn  101  -  Exp. 

20 

12 

26 

12 

2 

19 

6 

99 

All  Items 

4.  Form  101  -  Exp. 

17* 

9 

32 

12 

3.5 

21 

5 

57 

Items  know  by 

90+%  of  Group  of 

Full  Performance 
Level  (N*S0,  197$> 

5.  Form  101  -  Exp. 

19 

9 

28 

13 

2 

25 

4 

53 

Items  known  by  100% 
of  Group  of  Full 
Performance  Level 

ATC  (N-16,  1970) 

6.  Form  1018 

21 

12 

28 

12 

2 

19 

6 

103 

All  Items 

7.  New  Operational 

18 

10 

29 

13 

4 

21 

5 

100 

Forms  (Recoemended) 

Sub-elament:  1 

1 

• 

10 

9 

• 

5 

• 

2 

• 

2 

3 

2 

• 

• 

2 

3 

7 

2 

• 

1 

2 

11 

• 

4 

8 

1 

15 

• 

• 

* 

2 

5 

1 

1 

1 

1 

• 

0 

0 

6 

* 

3 

• 

2 

1 

7 

1 

1 
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TAPI  ! 


CATKlU'R IK-'-  tFl.FMKNTS  AND  Sl’R- ELEMENT  M  Of 
ATI  OOOITAT  U'NAl  KNi'WU  I>  I 


1.  AIK  TRAKEIC  KITTS 


01  . 

Pertaining 

t  O 

A  i  i  c  i  a  t  t 

0.'. 

Pert  a in ing 

tv* 

Alrvav 

(". 

Per  ta  mine 

t  O 

Air  space 

Oh. 

Per  f  a in i ng 

t  O 

Flving  Conditions  (Ratings,  Weather,  Speed 

Maneuvers , 

,  Altitude,  Flight  Level) 

OS. 

Pertaining 

to 

Opel  at  ions  Inst  rue  t 1 ons 

06. 

Per  t  aining 

to 

Administrative  Messages  (Reports,  Phraseology) 

O’. 

Pertaining 

to 

Sat  et  v  timer  gene  v 

2.  AIRPORT  TRAFFIC  PROCEDURES 

01  . 

Pertaining 

t  o 

Traffic  Information 

00. 

Pert  aining 

to 

Trail ic  Patterns 

03. 

Pertaining 

to 

Runva\  Ope rati on s 

Oh  . 

Pert  aining 

to 

Flight  Movement  (Destination  Changes,  Arrivals, 

Depart  tnent  s  ) 

OS. 

Pertaining 

to 

Administrative  Messages  (Phraseology,  Authorization, 

Official  Kept 

‘its.  Documents) 

06. 

Pert ain ing 

t  o 

Met erolog leal  Information  (Ceiling.  Visibility) 

O'. 

Pei  t  ainin.g 

t  o 

Safetv  (Messages,  Warning  Devices,  Collisions) 

».  INFLIGHT  TRAFFIC  CONTROL  PROCEDURES 

01  . 

Pertain  1 ng 

to 

Separation  (Vertical,  Lateral,  etc.) 

o:. 

Pertaining 

t  o 

Clearance  (Rout ing) 

O'. 

Pertaining 

to 

Approach  (Flight  Level,  Holding,  Alt.  Setting) 

Oh. 

Pert  aining: 

to 

Radar  (Contact,  ldent ,  Interference,  Traffic  Information) 

OS. 

Pertaining 

to 

Flight  Information  and  Data 

4.  COMMUNICATIONS  OPERATING  PROCEDURES 

01  . 

Pertaining 

to 

Flight  Movement  and  Control  Messages 

o:. 

Pertaining 

to 

Administrative  Messages  (Thrasoologv) 

03. 

Pertaining 

to 

Meterologv  Information 

04. 

Pertaining 

to 

Clearance 

OS. 

Pertaining 

to 

Safety 

06. 

Pertaining 

to 

Service /Maintenance 

5.  FLIGHT  ASSISTANCE  SERVICE  PROCEDURES 

01. 

Pertaining 

to 

Search  and  Rescue 

02. 

Pertaining 

to 

Overdue  Aircraft 

03. 

Pertaining 

to 

Emergency  Comnunl cat  ions 

04. 

Pertaining 

to  Meterological  Information 

OS. 

Pertaining 

to 

Facil i ties 

06. 

Pertaining 

to 

Procedures 

6.  AIR  NAVIGATION  AND  AIDS  TO  NAVIGATION 

to  Frequencies 
to  Instrumentation 
to  Posit ion/Dislance/Direct ion 
to  Navigation  (Time,  etc.) 
to  Navigational  Aids 

7.  AVIATION  WEATHER 
01.  Pertaining  to  Vis! hi  1 i ty/Cei 1 ing 

oi'  j’ertaJnlnK  t0  leather  Conditions  (Temperature,  Winds,  Tornado  etc  ) 
03.  Pertaining  to  Clouds  '  ' 

04.  Pertaining  to  Administrative  Messages  (Phraseology) 

03.  Pertaining  to  Forecasts/Reports 

06.  Pertaining  to  Responsibt 1 ltles/rrocedurrs 


01.  Pertaining 
02.  Pertaining 
03.  Pertaining 
04.  Pertaining 
03.  Pertaining 
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OKT  101B 

ITEMS  CODED  BY  ELEMENTS  AND  SUB-ELEMENTS 
(Underlined  Item  Numbers  Mean  90%  +  Right  for  FPL  C&L  -1975) 


r 


The  subject  matter  areas  or  elements  and  percentage  allocations  per  area 
as  determined  above  were  used  to  describe  the  types  and  number  of  items 
per  type  to  be  included  in  new  alternate  forms  of  the  test.  Fourteen  hundred 
multiple  choice  items  were  drafted  and  reviewed,  and  items  selected  from  this 
pool  to  create  eight  100-item  parallel  forms  of  the  test.  Twenty  items  were 
connnon  to  all  forms.  These  forms  were  administered  experimentally,  item 
analysis  accomplished,  and  the  eighty  best  items  in  each  form  identified  for 
keying.  These  eighty  item  forms  are  identified  as  ATC  Occupational  Knowledge 
Test  102A,  102B,  102C ,  102E,  102F,  102G,  and  102H.  Use  of  a  Rights  Only 
scoring  procedure  has  been  practiced  and  is  recommended . 

The  forms  have  been  administered  to  incoming  students  on  the  first  day  of 
training  at  the  ATC  Academy  since  November  21,  1978.  The  mix  of  trainees 
attending  the  Academy  varied  during  this  period  with  regard  to  levels  and 
types  of  aviation-related  experience.  In  order  to  control  these  differences, 
each  group  of  students  was  given  two  forms  of  the  Occupational  Knowledge 
test.  Consequently,  the  comparison  of  results  for  alternate  forms  of  OKI 
can  only  be  made  for  those  tests  taken  by  the  same  group  of  trainees. 

The  descriptive  statistics  obtained  for  each  trainee  group  which  took  two 
different  forms  of  the  test  are  provided  in  Table  *2. 

TABLE  42 

ALTERNATE  FORMS  OF  OKT 


<N=97) 


Mean 

SD 

Form  A 

41.95 

14.05 

Form  C 

42.55 

11.95 

(N=67) 

(N=22) 

Mean 

SD 

Mean  SD 

Form  A 

38.91 

14.61 

49.59  13.35 

Form  B 

36.25 

12.15 

44.73  10.84 

(N=86) 

Mean 

SD 

Foot  B 

38.35 

14.80 

Foot  D 

43.45 

12.75 

(N=384-386) 

(N=69-70) 

(N=22) 

Mean 

SD 

Mean  SD 

Mean  SD 

Foot  E 

46.49 

13.80 

45.51  13.98 

48.18  11.66 

Foot  F 

44.34 

13.40 

42.60  13.13 

44.50  12.10 

<N=407-408 

Mean 

SD 

Foot  G 

46.28 

13.95 

Foot  H 

48.38 

13.95 

Foot  G 
Foot  H 
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The  means  and  standard  deviations  for  the  two  forms  taken  by  each  group  can 
be  compared  since  the  experience  mix  for  that  group  was  common  across  both 
forms  taken.  However,  the  data  cannot  be  compared  for  the  same  form  taken  by 
different  groups  of  trainees  since  the  experience  mix  was  not  common. 

The  alternate  forms  do  show  some  differences  which  require  minor  adjustments 
by  shifting  items  of  a  common  type  but  ranging  in  difficulty  level, from  one  form 
to  another. 

Reliability  estimates  for  the  various  forms  have  been  derived  and  are  presented 
in  Table  3 .  These  reliabilities  are  considered  satisfactory  for  operational 
testing  use  and  administration. 


TABLE  43 


RELIABILITY  OF  ATC  OCCUPATIONAL  KNOWLEDGE  TEST 


Testing  Date 

Student  Groups 

OKT  102  Taken 

F irst  vs .  Second 

Cor  re  la 1 1 on 

3-6-7o 

Enroute 

N-86 

E.F.G.H 
(80  item) 

E.F.G.H 
(80  item) 

.909 

7-3-79 

Enroute  ,  Terminal^ 
FSS.  N-89 

E.F.G.H 
(80  item) 

E.F.C.H 
(80  item) 

.893 

Enroute,  Terminal 
N-183 

A.B.C.D 
(20  core 
items ) 

A.B.C.D 
(20  core 

items ) 

.877 

STUDY  OF  ATC  JOB  APPLICANTS  -  1978 


Objectives.  The  results  of  the  analysis  of  the  experimental  tests  for  a 
group  of  1827  new  ATC  trainees  at  the  FAA  Academy  during  the  period 
1976-1978  supported  the  conclusion  that  a  test  battery  comprised  of 
the  Multiplex  Controller  Aptitude  Test,  Abstract  Reasoning  (CSC-157) , 
and  Arithmetic  Reasoning  (CSC-24)  would  provide  an  operational  selection 
test  battery  for  the  ATC  occupation  with  significantly  better  prediction 
of  success  in  ATC  training. 

This  conclusion,  together  with  the  development  of  parallel  test  forms 
of  MCAT  made  it  desirable  to  administer  these  tests  to  a  new  group  of 
ATC  applicants  in  order  to  compare  the  results  of  the  proposed  test 
battery  with  the  existing  CSC  tests  with  respect  to  the  composition  of 
the  two  groups  who  "passed"  each  test  battery  and  the  effect  on  score 
distribution  within  each  group.  A  change  in  policy  also  made  it  possible 
to  obtain  race,  sex  and  ethnic  data  which  had  not  been  possible  with 
the  1976-1977  applicant  group. 

Arrangements  were  made  between  FAA  and  the  Office  of  Personnel  Management 
to  administer  the  parallel  forms  of  the  MCAT  and  a  60-item  version  of 
the  ATC  Occupational  Knowledge  Test  when  the  present  CSC  tests  were 
administered  during  September-November  1978  to  ATC  job  applicants. 

Sample  Description.  The  present  CSC  tests,  together  with  MCAT  and  OKT 

were  administered  to  a  total  of  6,000  ATC  job  applicants.  Wmv..-  5,331  applicants 

were  scheduled  for  examination  through  established  0PM  procedures.  In  addition, 

FAA  had  arranged  with  0PM  for  a  series  of  "walk-in"  test  sessions  which  did 

not  require  advanced  scheduling  by  applicants  with  0PM.  A  total  of 

669  applicants  took  the  test  on  a  "walk-in"  basis.  These  "walk-in" 

sessions  were  established  as  a  means  of  encouraging  more  women  and 

minorities  to  compete  for  positions  in  the  ATC  occupation.  Table  44 

provides  the  distribution  by  race,  ethnic  groups  and  sex  for  5295  of 

the  5331  "scheduled”  ATC  job  applicants.  Table  4  5  presents  these  data 


for  664 

of  the 

669  "walk- 

in"  applicants. 

Amer. 

Indian 

Asian 

Black 

TAH1.K  44 

White 

Hlsp- 

Other 

TOTAL 

Men 

42  (72%) 

38  (76%) 

650  (58%) 

2860  (76%) 

213  (79%) 

21  (70%) 

3824  (72%) 

Women 

16  (28) 

12  (24%) 

463  (42%) 

915  (24%) 

56  (21%) 

_9  (30%) 

1471  (28%) 

TOTAL 

58  (1%) 

50  (1%) 

1113  (21%) 

3775  (71%) 

269  (5%) 

30  (1%) 

5295  (100%: 

Amer 

Indian 

Aaian 

Black 

TAW. I.  4  5 

White 

Hi*p. 

Other 

TOTAL 

Men 

2 

3 

145  (50%) 

160  (55%) 

44  (65%) 

_ 

354  (53%) 

Women 

1 

4 

146  (50%) 

132  (45%) 

24  (24%) 

2 

310  (47%) 

TOTAL 

3 

7 

291  (44%) 

292  (44%) 

68  (10%) 

3 

664(100%) 
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There  are  several  points  of  interest  regarding  the  racial  and  sex 
distribution  of  these  two  applicant  groups.  For  the  "Scheduled"  applicant 
group,  (5295)  92  percent  were  black  or  white  men  and  women,  with  71  percent 
of  the  group  represented  by  white  men  and  women.  Generally  men  represented 
between  70-79  percent  of  each  racial  group  except  for  blacks;  only  58  percent 
were  men  while  42  percent  were  women;  almost  twice  the  ratio  of  other 
groups . 

For  the  "Walk-In"  applicants  the  distribution  indicates  t'\»c  the  objective 
of  having  more  minorities  and  women  compete  for  the  ATC  occupation  was 
fairly  successful.  Some  47  percent  of  the  total  group  were  women  in 
contrast  to  28  percent  for  the  scheduled  applicants  and  55  percent  were 
minorities  in  comparison  to  29  percent  of  the  scheduled  applicants. 

Predictors.  Tests  used  in  this  study  were  the  existing  CSC  selection 
test  battery  and  parallel  forms  of  MCAT  and  OKT: 


0PM  Test  Battel 


CSC-24  Arithmetic  Reasoning 
CSC- 51  Spatial  Relations 

CSC-135  Following  Oral  Direction 
CSC-157  Abstract  Reasoning 
CSC-540  Air  Traffic  Problems 


MCAT  0PM  ID  ODD/EVEN  TEST  COMBINATION 

.  Form  120  (4o6e) 

.  Form  130  (4e6o) 

.  Form  140  (6o7e) 

.  Form  150  (6e7o) 

Form  160  (7o4e) 

.  Form  170  (7e4o) 

OKT  -  101c  (60-item  version) 

As  in  previous  experimental  testing  of  applicant  groups,  the  MCAT  and 

OKT  test  were  administered  after  completion  of  the  regular  CSC  test  battery. 

In  addition  to  the  tests,  applicants  completed  forms  provided  by  OPM 
on  a  voluntary  basis  to  obtain  racial ,  sex,  and  other  information  for 
separate  analysis  by  OPM. 

Criterion  and  Analytical  Methods.  Since  this  study  encompassed  ATC  applicants, 
it  was  not  feasible  to  establish  operational  criterion  measures  of  validity. 
However,  a  number  of  statistical  analyses  were  made  based  on  pass/fail 
eligibility  comparison,  score  distributions  and  mean  and  standard  deviations 
on  the  various  tests. 
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Results .  Descriptive  statistics  for  the  "scheduled"  and  "walk-in"  groups 
were  developed  for  each  test.  It  should  be  pointed  out  that  of  the  260 
"Scheduled"  applicants  who  identified  themselves  as  "Hispanic",  all  but 
74  also  identified  themselves  with  another  racial  group.  For  example, 

139  were  identified  as  "Hispanic-White" .  In  this  analysis,  such  cases 
were  included  in  the  racial  group.  Consequently,  104  applicants  in  the 
"other"  category  shown  in  Table  43  were  comprised  of  74  Hispanic  applicants 
and  30  of  some  other  racial  groups  not  identified.  Descriptive  statistics 
for  the  "Scheduled"  group  of  applicants  are  provided  in  Table  46  and 
for  the  "Walk-In"  group,  in  Table  47.  (in  Table  44,  the  24  applicants 
in  the  "other"  category  consist  of  21  "Hispanic"  and  3  otherwise  not 
identified.) 


TABLE  46  SCHEDULED  ATC  APPLICANTS 


AMERICAN 
IMP  IAN 


(N- 

“66) 

IN.  59) 

(N-  1146) 

(N- 

3914) 

(N- 

T04) 

(N- 

"5331) 

Naan 

S.D. 

Naan 

S.D. 

Naan 

S.D. 

Maan 

S.D. 

Maan 

S.D. 

Maan 

S.D. 

CSC-24 

40.1 

9.2 

41.9 

7.9 

35.4 

9.6 

42.4 

7.8 

36.8 

8.6 

40.7 

8.9 

(Men) 

- 

- 

- 

- 

(34.3) 

(9.7) 

(42.2) 

(7.9) 

- 

- 

- 

- 

(Nqmh) 

" 

• 

- 

(37.0) 

(9.2) 

(43.2) 

(7.6) 

- 

- 

- 

" 

CS-51 

28.6 

4.8 

28.3 

5.8 

23.1 

6 .  S 

29.1 

5.2 

27.0 

6.5 

27.7 

6.1 

(Man) 

- 

- 

- 

- 

(24.3) 

(6.4) 

(29.6) 

(5.0) 

- 

- 

- 

- 

(WoMn) 

“ 

- 

- 

- 

(21.3) 

(6.2) 

(27.7) 

(5.6) 

- 

- 

“ 

CSC- 135 

21.6 

9.0 

19.9 

8.2 

16.6 

8.1 

24.7 

7.0 

20.1 

8.5 

22.7 

8.1 

(H.n) 

- 

- 

- 

- 

(16.7) 

(8.0) 

(24.8) 

(6.9) 

- 

- 

- 

- 

(tfoaan) 

“ 

“ 

- 

- 

(16.5) 

(8.3) 

(24.2) 

(7.4) 

- 

“ 

" 

- 

CSC-157 

28.0 

8.2 

31.0 

10.9 

22.1 

9.3 

30.9 

8.9 

26.1 

9.0 

28.8 

9 . 7 

<M«n) 

- 

- 

- 

- 

(21.5) 

(9.3) 

(30.5) 

(8.9) 

- 

- 

- 

- 

(tfoaan) 

“ 

- 

- 

- 

(22.9) 

(9.1) 

(32.1) 

(8.8) 

CSC- 540 

29.4 

12.7 

30.5 

12.6 

22.7 

11.9 

33.1 

11.7 

25.6 

12.5 

30.6 

12.5 

(Nan) 

- 

- 

- 

- 

(23.4) 

(11.9) 

(33.6) 

(11.6) 

- 

- 

- 

- 

(tfoaan} 

“ 

- 

(21.8) 

(11.9) 

(31.3) 

(11.9) 

- 

- 

HCAT 

31.1 

8.6 

31.0 

8.6 

22.1 

7.4 

34.2 

8.1 

28.0 

8.4 

31.4 

9.5 

(Man) 

- 

- 

- 

- 

(22.9) 

(7.8) 

(35.0) 

(8.0) 

- 

- 

- 

- 

(tfoaan) 

“ 

“ 

* 

- 

(21.1) 

(6.8) 

(31.8) 

(8.2) 

* 

_ 

- 

“ 

PICT 

(Man) 

(tfoaan) 

34.3 

10.8 

24.9 

11  •  3 

17.4 

8.7 

27.2 

11.7 

21.6 

11.8 

24.9 

11.8 

- 

- 

- 

- 

(11. B) 

(9.9) 

(28.8) 

(12.0) 

- 

- 

- 

- 

“ 

(15.5) 

(6.5) 

(22.0) 

(8.9) 

“ 

* 

— 

* 

' 

TABLE  47  WALK-IN 

ATC 

APPLICANTS 

AMERICAN 

INDIAN 

Jtttil 

ASIAN 

(N-  10) 

BLACK 

397) 

warn 

IS-  328) 

OTHER 
(H-  24) 

TOTAL 
<N-  669) 

Maan 

S.D. 

Main 

6.P. 

Maan  S.D. 

Maan 

8.P. 

Maan 

S.D. 

Maan 

S.D. 

CSC-24 

40.0 

4.2 

48.0 

9.5 

37.5  9.7 

42.3 

7.8 

39.4 

9.6 

40.1 

9.1 

CSC- 51 

39.0 

4.0 

31.8 

4.2 

23.0  6.2 

28.6 

5.0 

28.8 

4.2 

26.1 

6.3 

CSC- 13 5 

19.0 

7.2 

24.6 

7.3 

16.7  7.4 

23.7 

6.8 

19.7 

7.6 

20.4 

7.9 

CSC-157 

30.0 

11.6 

14.6 

12.2 

25.0  9.4 

32.0 

8.9 

28.7 

11.6 

2B.B 

9.9 

CSC -540 

36.4 

8.4 

45.2 

12.1 

23.6  12.7 

32.4 

12.5 

30.4 

13.4 

28.7 

13.5 

HCAT 

3S.4 

3.9 

34.9 

7.4 

22.7  7.5 

32.6 

7.8 

27.9 

9.7 

26.0 

9.1 

ORT 

19.2 

6.8 

16.4 

5.5 

16.2  6.5 

23.7 

10.9 

19.5 

8.9 

20.1 

9.7 

73 


For  the  "scheduled"  applicant  group,  the  mean  scores  for  women  were 
examined  for  both  the  "Black"  and  "White"  groups  to  assess  their  relation¬ 
ships  to  the  scores  of  men  in  the  same  racial  group.  Table  43  indicates 
that  women  score  somewhat  higher  on  CSC  tests  24  and  157  than  men  in 
each  racial  group,  but  somewhat  lower  on  CSC  51,  540,  MCAT,  and  OKT, 
with  the  greatest  difference  between  men  and  women  being  on  the  OKT. 

The  most  marked  difference  in  test  scores  between  racial  groups  is  for 
the  "Black"  and  "White"  groups,  with  the  mean  test  scores  for  "Blacks" 
generally  about  one  standard  deviation  below  the  mean  for  "Whites".  This 
holds  for  both  men  and  women  in  each  of  the  groups. 

Relatively  few  differences  between  mean  scores  of  the  "Scheduled"  and 
"Walk-In"  groups  are  evident  by  racial  groups.  Within  the  "Black"  group, 
walk-in  applicants  scored  about  2  points  higher  on  CSC  24  and  3  points 
higher  on  CSC-157.  The  "White"  group  walk-in  applicants  scored 
almost  2  points  lower  on  MCAT  and  about  3  points  lower  on  OKT.  These 
lower  scores  may  be  a  result  of  the  higher  percentage  of  women  in  the 
"White"  walk-in  group  (45  percent)  compared  to  the  scheduled  group 
(24  percent) . 

With  the  applicant  group  tested  in  1976-1977,  two  of  the  CSC  Tests  (51 
Spatial  Relations  and  135  Following  Oral  directions)  showed  marked  negative 
skew.  It  was  of  interest  to  determine  if  this  characteristic  was  replicated 
for  these  two  tests  with  the  1978  applicant  population.  Figure  4  provides 
a  frequency  distribution  for  test  scores  for  these  two  CSC  tests.  It 
is  evident  that  again  they  provide  little  differentiation  between  the 
applicants. 

FIGURE  4 


Veterans  preference  for  the  total  group  (scheduled  and  walk-in)  was  identified 
by  sex  and  racial  groups.  Table  48  provides  the  distribution  by  sex 
for  5846  of  the  6Q00  applicants. 

TABLE  48 

SJEX  DISTRIBUTION-]  9  7  6-1977  APPL  I  CANT  GROUP 


VET 

PREF  MEN  WOMEN  TOTAL 


None 
5  Pt. 

10  +  Pt. 


2406  (59%) 
1571  (38%) 
123  (  3%) 


1640  (94%) 

9$' 6%) 


4046  (69%) 
1668  (29%) 
132  (  2%) 


TOTAL 


4100  (100%)  1746  (100%) 


5846  (100%) 


Table  49  provides  the  distribution  of  Veterans  Preference  points  by  racial 
and  ethnic  group. 

TABLE  49 

VETERANS  PREFERENCE  DISTRIBUTIONS 


VET. 

AMERICAN 

PREF. 

INDIAN 

ASIAN 

BLACK 

WHITE 

HISP. 

OTHER 

TOTAL 

None 

42  (70%) 

40  (70%) 

981  (73%) 

2724  (68%) 

223  (68%) 

27  (84%) 

4037 

(69%) 

5PTS. 

10  +  PTS. 

17j(30%)  • 

17^(30%) 

33<27'> 

12$<32%> 

99j(32%) 

fj(16%) 

1666 

132 

(29%) 

(2%) 

TOTAL 

60 

57 

1340 

4017 

329 

32 

5835 

Since  Veterans  Preference  points  are  added  to  the  test  battery  score  for 
those  who  pass  the  test,  it  is  evident  that  only  6  percent  of  the  women  could 
benefit,  and  thus  be  ranked  higher  on  the  0PM  register  for  appointment  eligi¬ 
bility.  This  in  contrast  to  41  percent  of  the  men  who  could  benefit  from 
military  service. 

Eligibility  for  Veterans  Preference  points  by  racial  and  ethnic  groups  is  about 
evenly  distributed  (30-32  percent)  except  for  the  "Black"  group  (27  percent) 
where  the  higher  ratio  of  women  applicants  to  men  reduces  the  number  who  receive 
Veterans  Preference  points. 
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Table 

50  compares  the  total 

ATC  applicant  group 

to  the  civilian 

force 

(CLF)  by  racial  and  ethnic  group  based  on 

1970  census  data 

amendments  through  1977. 

tabu:  so 

ATC  API’ LI 

[  CANTS-C 1 V I L 1  AN  LABOR 

FORCE 

Civilian 

Labor 

%  ATC 

Force 

Applicants 

American  Indian 

.3% 

1% 

Asian 

.8% 

1% 

Black 

9.7% 

24% 

White 

85.0% 

68% 

Hispanic 

4.2% 

6% 

In  terms  of  ATC  applicants,  blacks  as  a  group  are  significantly  over¬ 
represented  in  comparison  to  their  percentage  of  the  civilian  labor 
force  while  whites  as  a  group  are  underrepresented.  As  a  group,  women 
were  also  underrepresented;  they  comprised  28  percent  of  the  ATC  applicant 
group  compared  to  38  percent  of  the  civilian  labor  force. 

Analysis  of  current  PPM  Test  and  OKT 

Next,  the  current  0PM  test  battery  scores  were  analyzed  in  terms  of 
"pass/fail"  score  distribution  by  ethnic  or  racial  group  and  by  men  and 
women.  Because  of  the  small  number  of  Asian  and  American  Indian  applicants, 
analysis  of  minority  groups  has  been  directed  to  Blacks  and  Hispanics. 

Table  51  provides  the  results  of  those  who  took  the  current  0PM  ATC  test 
with  respect  to  "pass/fail".  The  minimum  passing  score  on  the  test 
battery  is  70  which  equates  to  the  approximate  mean  score  on  the  test 
where  about  half  of  the  total  applicants  pass  and  half  fail. 


1AR1.F  51 

PASS/ FA  1  l.-Oi'M  ATC  TEST 

Pass 


Total 

0PM 

Test 

0PM  Test 

Group 

Applicants 

N 

_% 

N 

%, 

Men 

4191 

2236 

(53%) 

1955 

(47%) 

Women 

1785 

799 

(45%) 

986 

(55%) 

Total 

5976 

3035 

(51%) 

2941 

(49%) 
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Compared  to  the  7500  applicants  tested  in  1976-1977,  this  group  had  about 
the  same  pass  rate  for  men  (53  percent  compared  to  52  percent)  and  a  some¬ 
what  higher  pass  ratio  for  women  (45  percent  compared  to  41  percent). 

Table  52  shows  the  "pass/fail"  rates  by  racial  group. 


TABU.  52 

PASS/FAIL  BY  RACIAL  CROIT 


Group 

Total 

Applicants 

Pass 

N 

Fail 

N 

% 

White 

4067 

2556 

(63%) 

1511 

(37%) 

Hispanic 

339 

128 

(38%) 

211 

(62%) 

Black 

1407 

264 

(19%) 

1143 

(81%) 

Total 

5813 

2948 

(51%) 

2865 

(49%) 

In  view  of  the  disproportionate  number  of  minorities  who  did  not  pass 
the  OPM  test,  an  analysis  of  score  distribution  was  made  by  racial 
group  (Table  53) . 

TABLE  53 

OPM  TEST  SCORE  DISTRIBUTION  BY  RACIAL  CROUP 


Fail  or*  T«it 

(N  -  2865) 


Group 

50 

in 

O 

59 

60-69 

i 

70-79 

80- 

89 _ 

_ 

89 

Total 

[ 

N 

N 

» 

H 

% 

N  % 

1 

H  % 

% 

N 

% 

Whits 

185 

(«) 

406 

(10%) 

920  (23X) 

1 

1128  USD 

975 

(24%) 

453 

(11%.) 

4067 

Hispanic 

63 

(181) 

56 

(17%) 

92  (272) 

1 

1 

70  (21%) 

41 

(12%) 

17 

(5%) 

339 

•lack 

457 

021) 

381 

(27%) 

305  (222) 

1 

189  (1 32) 

63 

(  5%) 

12 

(1%) 

1407 

Pass  OPM  Test 

(N  -  29 <.8) 


It  is  evident  from  this  analysis  that  for  the  Black  group  in  particular, 
the  higher  fail  rate  is  comprised  of  a  large  number  of  applicants  who  did 
very  poorly  on  the  test  battery.  The  457  Blacks  who  scored  below  50  on 
the  test  comprised  40  percent  of  the  total  number  who  failed  to  pass  the 
test  in  contrast  to  about  30  percent  for  Hispanic  and  12  percent  for  Whites. 
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Figure  5  provides  a  graph  of  the  OPM  ATC  test  score  distribution  for 
selected  racial  groups. 


FIGURE  5 


mrsavce* 


Table  54  compares  the  distribution  of  ATC  applicants,  ATC  applicants  who 
passed  the  OPM  test  and  the  Civilian  Labor  Force  (CLF) . 

TABLE  54  _ 


CIVIL  LABOR 

FORCE- ATC “ 

APPL 1  CAM'S -PA S  S  CR01 ' 

T 

Civ.  Labor 

%  of  % 

of  Applicants 

Force 

ATC  Applicants  in  Pass  Group 

American  Indian 

.3% 

1% 

1.0% 

Asian 

.8% 

1% 

1.1% 

Black 

9.7% 

24% 

8.9% 

White 

85.0% 

68% 

84.8% 

Hispanic 

_ iJH 

6% 

4.2% 

100.0% 

100.0% 

100.0% 
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Administration  of  the  ATC  Occupational  Knowledge  Test  (OKT)  provided  the 
first  opportunity  to  evaluate  the  results  for  a  sizable  sample  of  ATC 
applicants.  The  OKT  used  was  version  101  C,  a  60-item  test.  Consequently, 
the  results  of  this  test  were  reviewed  for  the  total  group,  men,  women, 
racial  and  ethnic  groups  and  relationship  of  the  results  to  those  applicants 
with  veterans  preference. 

Tables  43  and  44  provided  means  and  standard  deviations  on  the  OKT  Test 
results  for  the  various  groups.  The  "walk  in"  group  of  applicants  had  a 
lower  mean  score  than  the  "scheduled"  group;  20.1  in  comparison  to  24.9  and 
this  difference  was  evident  in  all  racial  groups  except  for  blacks  where 
the  mean  score  was  17.4  (scheduled)  and  16.2  (walk  in).  This  would  indicate 
that  there  was  relatively  little  difference  in  ATC  related  knowledge  for 
blacks  of  either  group  but  that  other  applicants  in  the  "Walk  In"  group  had 
less  ATC  related  knowledge  or  experience  that  those  in  the  "Scheduled" 
group.  In  view  of  the  objective  of  the  "Walk  In"  recruitment  program,  these 
differences  would  be  expected. 

Table  55  provides  the  descriptive  statistics  for  men  and  women  on  OKT. 

TABLE  55 

DESCRIPTIVE  STATISTICS,  SEX  VS.  OKT 

Scheduled  Walk  In 


Group 

(N) 

Mean 

SD 

(N) 

Mean 

SD 

Men 

(3835) 

26.9 

12.2 

(356) 

22.2 

11.2 

Women 

(1473) 

19.8 

8.7 

(312) 

17.6 

6.9 

Table  56  provides  the  OKT  score  distribution  for  men  and  women  and  for  racial 
groups.  In  deriving  this  distribution,  the  raw  scores  were  converted  to  a 
scale  of  0-100.  In  this  analysis,  the  Hispanic  group  was  distributed  within 
the  racial  groups  shown. 

TABLE  56 

Distribution  of  OKT  Scores 


Group 

N 

Below 

15 

15-24 

25-34 

Men 

(4191) 

3% 

11% 

27% 

Women 

(1785) 

5% 

20% 

43% 

White 

(4242) 

2% 

9% 

27% 

Asian  j 

American  Indian] 

|  (  140) 

3% 

14% 

35% 

Black 

(1443) 

8% 

32% 

39% 

35-44 

45-54 

55-64 

65-74 

75-79 

Above 

79 

20% 

9% 

10% 

9% 

8% 

3% 

19% 

5% 

4% 

3% 

1% 

* 

22% 

10% 

10% 

9% 

7% 

3% 

18% 

9% 

9% 

6% 

5% 

1% 

13% 

2% 

2% 

2% 

2% 

* 

*  Less  than  1% 
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Approximately  20  percent  of  the  men  scored  above  64  as  compared  to  4  percent 
of  the  women;  19  percent  of  the  white  group;  12  percent  of  the  Asian/ 

American  Indian;  and,  4  percent  of  the  black  group  scored  above  64  respectively. 


Previous  studies  with  ATC  trainees  at  the  Academy  as  well  as  developmental 
and  Journeymen  ATC  specialists  provided  a  significant  correlation  between 
scores  on  the  OKT  test  and  success  in  ATC  training  and  work.  These  studies 
also  supported  the  conclusion  that  the  OKT  provided  a  better  basis  for 
granting  extra  credit  to  ATC  applicants  who  passed  the  0PM  ATC  Test  them 
the  present  rating  guide  used  for  this  purpose.  Consequently,  for  further 
analysis,  this  applicant  group's  OKT  scores  were  given  the  following  "extra 
points" : 


OKT  Scores 

65-69 

70-74 

75-79 

80+ 


Extra  Credit 

3  points 
5  points 
10  points 
15  points 


These  point  values  are  similar  to  those  presently  provided  by  the  0PM  rating 
guide  for  specific  types  of  ATC  related  experience  which  now  provides  5,  10 
or  15  points  depending  on  the  specific  experience. 

In  most  cases,  the  types  of  ATC  related  experience  for  which  an  applicant 
is  presently  granted  extra  credit  is  obtained  through  military  service 
and  training.  For  example.  Air  Traffic  Control  experience  which  is  granted 
the  most  points  (10-15)  is  obtained  through  military  experience.  This  also 
tends  to  be  the  case,  although  to  a  lesser  extent,  for  pilot  and  other  types 
of  credited  experience. 


In  order  to  assess  the  relationship  between  military  experience  and  OKT 
scores,  an  analysis  of  those  applicants  with  veterans  preference,  their 
OKT  scores,  and  the  extent  to  which  they  could  earn  extra  credit  was 
completed.  The  results  are  provided  in  Table  57  below. 

'I  ABLE  57 


Veterans  Preference 
No  Yes 


Score 

Credit 

N 

% 

N _ 

_ % 

Total 

65-69 

3  pts 

103 

(45%) 

126 

(55%) 

229 

70-74 

5  pts 

76 

(33%) 

153 

(67%) 

229 

75-79 

10  pts 

61 

(24%) 

190 

(76%) 

251 

80+ 

15  pts 

26 

(10%) 

233 

(90%) 

259 

266 

(28%) 

702 

(72%) 

968 

80 


Prior  analysis  of  the  relationship  between  OKT  scores  and  "pass/fail"  of 
ATC  training  for  1827  students  indicated  that  the  largest  drop  in  total 
failure  rate  was  between  those  trainees  who  scored  between  60-64  '29.6 
percent  failure  rate)  and  65-69  (20.1  percent  failure  rate) .  Consequently, 
for  this  analysis  comparisons  were  based  on  providing  3  points  for  those 
applicants  who  scored  between  65  and  69.  About  16  percent  (968)  of  the 
total  6000  applicants  could  have  been  eligible  for  additional  point  credit 
based  on  OKT  test  scores.  A  total  of  702  (72  percent)  also  could  receive 
points  for  veterans  preference.  The  increasing  number  of  applicants  with 
veterans  preference  who  scored  "highest"  on  the  OKT  indicate  that  most  of 
them  would  probably  have  had  military  air  traffic  control  experience. 

In  contrast,  those  applicants  who  were  not  veterans  constituted  28  percent 
of  the  total  group  who  could  receive  extra  credit  based  on  OKT  scores. 
While  applicants  in  this  group  score  "lower"  on  OKT,  use  of  this  test  to 
grant  extra  credit  would  broaden  the  opportunity  for  non-veterans  who 
acquired  ATC  related  knowledge  through  other  experience  to  also  qualify  for 
extra  credit. 

Under  current  OPM  rating  and  ranking  procedures,  an  applicant  must  make  a 
passing  score  of  at  least  70  on  the  present  ATC  test  battery  before 
additional  credit  is  given  for  veterans  preference  or  aviation  related 
experience.  Of  the  total  968  applicants  who  could  be  eligible  for  extra 
credit  based  on  OKT  scores,  272  (28  percent)  scored  below  70  on  the  present 
OPM  ATC  test  battery.  Table  58  shows  the  distribution  by  "point  groups”. 

TABLE  58 

ELIGIBILITY,  VETERANS  PREFERENCE,'  AV I  AT  10 N  RELATED  EXTO1 ENCE 


Extra 

Points 

Total 

Failed  OPM 
ATC  Test 

Total 

Eligible  for 
Extra  Credit 

15  pts 

259 

58 

(22%) 

201 

(78%) 

10  pts 

251 

74 

(29%) 

177 

(71%) 

5  pts 

229 

62 

(27%) 

167 

(73%) 

3  pts 

229 

78 

(34%) 

151 

(66%) 

968 

272 

(28%) 

696 

(72%) 

The  696  applicants  who  passed  the  OPM  ATC  test  and  would  be  eligible  for 
extra  credit  based  on  OKT  scores,  represented  23  percent  of  the  3083  appli¬ 
cants  who  passed  the  test. 

Next  the  data  were  examined  in  terms  of  the  ATC  test  score  distribution  and 
the  effect  of  adding  additional  credit  for  veterans  preference  and  aviation 
related  experience  on  the  score  distribution  for  those  who  passed.  Figure  6 
provides  the  distribution  of  raw  OPM  test  scores  for  the  total  applicant 
group. 
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Figure  7  provides  the  sc 
the  OPM  test  after  addin 
scores. 


It  is  evident  that  grant: 
significantly  shifts  the 
899  applicants  scoring  9< 
to  500  n^r,r,#anf-  of  t-Vw 


U  rr«  j/74*w  J/tf  SO 


Figure  8  shows  the  distribution  of  OPM  scores  after  adding  extra  credit 
for  veterans  preference  and  OKT  test  scores  for  men  and  women. 


FIGURE  8 
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While  a  significant  shift  to  higher  scores  is  apparent  for  men,  there  is 
relatively  little  change  for  women.  Of  the  two  extra  credit  factors, 
veterans  preference  accounts  for  more  of  the  shift  than  OKT  credit. 

For  every  woman  who  receives  extra  credit  for  being  a  veteran,  16  men 
also  get  veteran's  preference  credit.  By  comparison,  for  every  woman 
who  would  get  extra  credit  for  aviation-related  experience  based  on  OKT 
scores,  9  men  would  also  get  extra  credit.  While  both  factors  result  in 
moving  more  men  than  women  to  higher  total  score  ranges  for  appointment 
consideration,  almost  twice  as  many  men  are  moved  up  based  on  veterans 
preference  than  on  OKT  scores. 

The  effect  of  granting  extra  credit  on  score  distribution  can  also  be 
examined  in  terms  of  the  relative  representation  of  men,  women  and  minority 
groups  in  various  score  ranges  with  and  without  the  extra  credit. 

Table  56  shows  the  percentage  of  men  and  women  in  each  of  four  score  range 
groups  based  on  raw  OPM  test  scores  (before  veterans  preference  or  OKT 
extra  credit  is  applied)  and  compares  those  to  the  percentage  of  men  and 
women  in  each  group  after  applying  extra  credit.  It  should  be  noted  that 
by  adding  extra  credit  it  would  be  possible  to  have  a  maximum  total  score 
which  exceeds  100. 
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TABLE  59 


PPM  TEST  SCORE  CROUPS 
(Pass  Sample  N=3035) 


*->1 

o 

1 

79 

00 

? 

00 

KO 

90-100 

90-99 

100+ 

Croup 

Raw 

Score 

(N=1440) 

With 

Credit 

(N=1076) 

Raw 

Score 

(N=1106) 

With 

Credit 

(N=1086) 

Raw 

Score 

(N=489) 

With 

Cred  i  t 
(N=644) 

With 
Credit 
( N= 229 ) 

Men 

74% 

67% 

73% 

73% 

73/ 

791 

89 

Women 

26% 

33% 

27% 

27%. 

211 

211 

11,' 

100% 

100% 

100% 

100% 

100% 

1001 

100, 

Based  on  the  raw  OPM  test  scores,  women  comprise  26-27  percent  of  each  of  tilt- 
score  range  groups  as  compared  to  men.  Since  women  comprised  26  percent  of 
the  total  pass  group,  they  are  also  proportionately  represented  (as  compared 
to  men)  in  each  of  the  score  range  groups.  However,  when  veterans  preference 
and  OKT  credit  is  included,  the  percentage  of  women  increased  to  33  percent 
in  the  70-79  score  group;  remains  the  same  in  the  80-89  group;  and  drops  to 
21  percent  and  11  percent  in  the  two  “highest"  scoring  groups.  Table  .59 
provides  a  quantative  measure  of  the  extent  to  which  veterans  preference  and 
aviation  related  experience,  as  measured  by  OKT,  differentially  affect  men 
and  women  for  appointment  consideration. 

Table  60  compares  the  same  data  by  minority  groups. 

TABLE  60 

OPM  TEST  SCORE  GROUPS 
(Pass  Sample  N=2948 


70- 

-79 

80- 

-89 

90-100 

90-99 

100+ 

Group 

Rating 

(N=1387) 

Augmented 

Rating 

(N=1042) 

Rating 

(N=1079) 

Augmented 

Rating 

(N=1058) 

Rating 

(N=482) 

Augmented 

Rating 

(N=622) 

Augmented 
Rat ing 
_(N= 2  26) 

White 

81% 

80% 

90% 

89% 

94% 

93% 

937 

Hispanic 

5% 

5% 

4% 

4% 

4% 

4% 

>4  / 

Black 

14% 

15% 

6% 

7% 

2% 

_ 3% 

L" 

100% 

100% 

100% 

100% 

100% 

100% 

ioo; 

In  the  case  of  Black  and  Hispanic  minority  groups,  it  is  evident  that  granting 
veterans  preference  and  OKT  credit  has  very  little  impact  on  the  proportionate 
representation  within  each  OPM  score  group  for  those  applicants  who  pass  the  Ol’M 
test.  Overall,  Table  60  indicates  that  veterans  preference  and  credit  for 
aviation  experience  affects  each  of  the  racial  groups  essentially  the  same  with 
respect  to  their  score  distribution. 
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Analysis  o£  Experimental  ATC  Test  and  OKT 

Based  on  prior  research  conducted  by  the  Civil  Aeromedical  Institute  on 
a  group  of  1827  ATC  trainees  at  the  FAA  Academy,  it  was  shown  that  an 
experimental  ATC  test  battery  comprised  of  the  Multiplex  Controller 
Aptitude  Test  (MCAT) ,  the  OPM  Test  157  -  Abstract  Reasoning,  and  the 
0PM  Test  24  -  Arithmetic  Reasoning  provided  a  high  multiple  correlation 
(R=. 54)  with  ATC  training  laboratory  scores  (19).  It  was  also  shown 
that  the  use'  of  an  ATC  Occupational  Knowledge  test  added  significantly 
to  the  statistical  correlation  (  2 f> )  ( /  > . 

Since  parallel  forms  of  MCAT  and  a  60-item  version  of  OKT  (101  C)  were 
administered  together  with  the  current  OPM  ATC  test,  it  was  possible  to 
construct  an  Experimental  ATC  Test  Battery  comprised  of  MCAT,  Abstract 
Reasoning,  Arithmetic  Reasoning,  and  OKT  and  compare  these  results  to 
the  current  OPM  test  battery  for  the  same  6000  applicant  group. 

Consequently,  the  analyses  previously  discussed  for  the  OPM  ATC  test  were 
also  completed  using  this  Experimental  ATC  test  battery.  Each  of  the 
three  tests  (excluding  OKT)  were  equally  weighted  in  deriving  a  composite 
score  for  the  experimental  ATC  test  battery. 

The  table  numbers  used  in  this  section  will  be  the  same  as  those  used  in 
the  OPM  test  analysis  followed  by  a  suffix  "E"  where  the  same  data  is 
presented  for  the  Experimental  ATC  test. 

Table  hi  provides  the  results  of  using  the  Experimental  ATC  test  with 
respect  to  "pass/fail".  A  minimum  raw  score  of  100.47  was  used  which  also 
equates  to  the  mean  score  on  the  test  battery  where  about  half  of  the 
applicants  pass  and  half  fail. 

I AHLK  61 

I’ASS/l’Alj.  -  IXl’liK  I  MENTAL  ATC  TEST 

PlM  Pail 


Group 

local 

Applicants 

Exp.  ATC 
N 

Test 

in 

1*2-. 

N 

ATC  Test 

in 

Hen 

4191 

2312 

(551) 

1879 

(451) 

1785 

852 

(48*) 

933 

(52:) 

Total 

5976 

3164 

(53J) 

2812 

(471) 

With  the  Experimental  ATC  test  battery  (excluding  OKT),  a  somewhat  higher 
percent  of  the  total  applicants  passed  compared  to  the  OPM  test  (53  percent 
compared  to  51  percent). 

With  respect  to  sex  groups,  both  men  and  women  passed  the  Experimental  ATC 
test  at  a  higher  rate,  (55  percent  men  compared  to  53  percent  on  the  OPM 
test  and  48  percent  women  compared  to  45  percent  with  the  OPM  test).  However, 
the  mean  raw  score  on  the  Experimental  ATC  test  was  used  as  a  "pass”  score. 

On  the  current  OPM  ATC  test  the  mean  raw  score  for  this  applicant  group 
was  207.1  compared  to  the  established  pass  score  of  210.  Consequently,  the 
"higher"  pass  rate  is  primarily  a  result  of  the  pass  scores  utilized. 
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Given  this  general  comparison  of  the  two  tests,  one  key  point  then  is 
the  extent  to  which  the  individual  applicants  who  passed  the  two  tests 
in  fact  differ.  Table  62  shows  the  number  of  applicants  who:  (1)  passed 
both  tests;  (2)  failed  both  tests;  (3)  passed  the  OPM  test  but  failed  the 
Experimental  ATC  test;  or  (4)  failed  the  OPM  test  but  passed  the  Experi¬ 
mental  ATC  test. 

TAB1.K  62 

TEST  RESULT  DIFFERENCES"-  INDIVIDUAL  APPLICANTS 


Passed  Both 

Failed  OPM/Passed  ATC 

(1) 

(4) 

2727 

437 

(46%) 

(7%) 

Passed  OPM/Failed  ATC 

Failed  Both 

(3) 

(2) 

308 

2504 

(5%) 

(42%) 

Pass  Exp. 
ATC  3164 


Fail  Exp. 
ATC  2812 


Pass  OPM 
3035 


Fail  OPM 
2941 


TOTAL 

5976 


From  Table  62  it  is  evident  that  using  the  two  different  tests  affected  the 
pass/fail  itatus  of  745  (12  percent)  of  the  total  applicants.  In  terms  of 
the  Experimental  ATC  test,  437  different  applicants  (14  percent  of  those  who 
passed)  would  be  eligible  for  appointment  consideration  who  would  not  be 
eligible  based  on  the  current  OPM  test.  While  most  of  the  shifts  between 
pass/fail  on  the  two  tests  occur  around  the  passing  score  of  70,  some 
applicants  who  scored  as  low  as  45  on  the  OPM  test,  passed  the  ATC  Experimental 
test.  Conversely,  some  applicants  who  scored  as  high  as  85  or  the  OPM  test 
failed  the  ATC  experimental  test. 

Table  63  shows  the  pass/fail  rates  on  the  Experimental  ATC  test  by  racial 
groups . 
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PASS /FAIL  RATES  -  EXPERIMENTAL  RACIAL  CROUPS 


Total 

E*P- 

ATC  Test 

ATC  Test 

Croup 

Applicants 

N 

in 

S 

in 

Whit* 

4067 

2678 

(66X) 

1389 

(342) 

Hispanic 

339 

141 

(421) 

198 

(582) 

Black 

1407 

256 

li«L 

1151 

18221 

Total 

5813 

3075 

(532) 

2738 

(472) 

For  the  Experimental  ATC  test,  the  percentage  of  the  White  group  who  passed 
was  somewhat  higher  than  on  the  OPM  test  (66  percent  compared  to  64  percent). 
The  Hispanic  group  also  increased  in  pass  rate  (40  percent  compared  to  38 
percent)  and  the  Black  group  pass  rate  decreased  slightly  (18  percent 
compared  to  10  percent). 

Again,  because  of  the  disproportionate  number  of  minorities  who  did  not  pass 
the  Experimental  ATC  test,  analysis  of  score  distribution  is  provided  in 
Table  64. 

TABLE  64 

SCORE  D I STR1 BUT  1 0NS-EXPER1 MENTAL  ATC  TEST 

(N-2738)  (N-3075) 


Fall  Exp  ATC 

Test  — 

- > 

-  Pass 

Exp.  ATC  Test 

Below 

i 

Croup 

50 

SO- 

-59 

60- 

69  ! 

70- 

79 

80-89 

89 

»  in 

li 

ill 

N 

ill  ! 

1 

N 

ill 

N  ill 

n  m 

Total 

hlte 

39  (  12) 

329 

(  82) 

1021 

(252)  ; 

1758 

(4  32) 

857  (212) 

63  (  22) 

406  7 

ispanlc 

23  (  72) 

74 

(212) 

101 

(302)  j 

105 

(312) 

33  (102) 

3  (  12) 

339 

Lack 

245  (172) 

465 

(332) 

441 

(322)  ; 

1 

227 

(162) 

28  (  22) 

1  (  *  ) 

1407 

It  is  evident  from  this  analysis  that,  for  all  racial  groups,  the  number 
of  applicants  who  did  very  poor  on  the  Experimental  ATC  test  (scores  below 
50)  is  much  smaller  than  on  the  OPM  test.  For  example,  32  percent  of  the 
Black  applicants  scored  below  50  on  the  OPM  test  compared  to  17  percent 
on  the  Experimental  test.  This  also  held  true  for  applicants  who  scored 
very  high  on  the  OPM  test  where  11  percent  of  the  White  group  scored  above 
89  in  contrast  to  only  2  percent  on  the  Experimental  ATC  test. 

Figure  9  provides  a  graph  of  the  Experimental  ATC  test  score  distribution 
for  selective  racial  groups. 
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FIGURE  9 


Table  65  shows  a  comparison  of  the  distribution  of  applicants  who  passed 
the  Experimental  ATC  test  with  those  who  passed  the  OPM  test,  and  the 
distribution  of  ATC  applicants  with  the  Civilian  Labor  Force  (CLF) . 

TABU'.  65 

EXPERIMENTAL  ATC  TEST  -  OPM  IT.ST  RESULTS 


Group 

Civilian 
Labor  Force 

%  of  ATC 
Applicants 

%  of 

OPM  Test 

Pass  Group 

%  of  ATC 
Exp.  Test 
Pass  Group 

Am.  Ind. 

.3% 

1% 

1.0% 

1.2% 

Asian 

.8% 

1% 

1.1% 

1.3% 

Black 

9.7% 

24% 

8.9% 

7.6% 

White 

85.0% 

68% 

84.8% 

85.5% 

Hispanic 

4.2% 

6% 

4.2% 

4.4% 

100.0% 

100.0% 

100.0% 

100.0% 

In  analyzing  the  relationship  of  the  Experimental  ATC  Tert  and  the  Occu¬ 
pational  Knowledge  Test  (OKT) ,  the  discussion  and  descriptive  statistics 
for  the  total  applicant  group,  as  provided  in  Table  55,  56,  and  57  also  apply 
here. 

However,  some  applicants  who  passed  the  OPM  Test  failed  the  Experimental 
ATC  Test  (and  vice  versa) .  Consequently,  the  number  of  applicants  (and 
individual  applicants)  who  could  be  eligible  for  extra  credit  based  on 
OKT  scores  will  differ  for  the  two  tests. 

Table  66  provides  the  number  and  distribution  by  "point  groups"  of  the 
total  applicant  group,  those  who  failed  the  Experimental  ATC  Test,  and 
those  who  would  be  eligible  for  extra  credit  based  on  their  OKT  scores. 


TABLE  66 

TOTAL  APPL  1 CAN T_  TO  I  NT  CK OCl'S 

Total 


Extra 

Failed  Exp. 

Eligible  for 

Points 

Total 

ATC 

Test 

Extra 

Credit 

15  pts. 

259 

60 

(23%) 

199 

(77%) 

10  pts. 

251 

70 

(28%) 

181 

(72%) 

5  pts. 

229 

59 

(22%) 

170 

(78%) 

3  pts. 

229 

85 

(37%) 

144 

(63%) 

968 

274 

(28%) 

694 

(72%) 

Of  the  total  968  applicants  who  could  be  eligible  for  extra  OKT  credit, 
a  total  of  206  failed  both  the  OPM  and  the  Experimental  test.  A  total 
of  66  failed  the  OPM  test  but  passed  the  Experimental  ATC  test.  Con¬ 
versely,  68  passed  the  OPM  test  but  failed  the  Experimental  ATC  test. 
Consequently,  the  two  tests  affected  340  applicants  or  35  percent  of  the 
applicants  who  scored  above  64  on  the  CKT. 

Figure  10  provides  the  distribution  of  raw  Experimental  test  scores  for 
the  total  applicant  group  in  comparison  to  the  OPM  test  score  distribution. 
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FIGURE  I*' 


TOTAL  ATC  APPLICANT  GROUP 
(N  =  6000) 


Those  applicants  who  scored  below  45  on  the  OPM  ATC  test  tend  to  cluster 
in  the  50-59  score  range  on  the  Experimental  ATC  test.  Table  67  also 
provides  the  comparative  distribution  statistics  for  the  "pass  group" 
for  each  of  the  tests. 


TABLE  67 


PASS  GROUP 

TEST 

SCORE 

DISTRIBUTION 

70- 

21 

75-79 

80 

-84 

85-89 

90- 

94 

95* 

H 

_ 

H 

_ %_ 

N 

% 

N  _ %_ 

N 

J _ 

N 

% 

TOTAL 

OPM  T«lt 

697 

(23X) 

743 

<2«) 

635 

(21X) 

471  (16X) 

310 

(101) 

179 

(6X) 

3035 

Exp.  ATC  T«at 

1253 

(40X) 

895 

(2  8X) 

606 

U9X) 

341  (11X) 

63 

(  2X) 

6 

(*) 

3164 

The  Experimental  ATC  test  provides  a  considerably  greater  differentiation  on  test 
scores  between  the  applicants  who  pass  as  compared  to  the  OPM  test  with  a  much 
higher  clustering  of  applicants  in  the  70-79  score  range  (67  percent  com¬ 
pared  to  48  percent)  and  a  much  smaller  number  of  applicants  with  scores 
of  90  and  above  (2  percent  compared  to  16  percent) . 
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The  Experimental  ATC  test  data  were  also  examined  in  terms  of  test  score 
distribution  and  the  effect  of  additional  credit  for  veterans  preference 
and  aviation-related  experience  on  the  score  distribution  for  those  who 
passed. 

Figure  1 !  provides  the  score  distribution  for  all  ATC  applicants  who 
passed  the  Experimental  ATC  test  after  adding  extra  credit  for  veterans 
preference  and  OKT  test  scores  above  64. 

FIGURE  1  1 

EXPERIMENTAL  ATC  TEST 
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With  the  Experimental  ATC  test,  a  total  of  485  applicants  scored  90  or 
above  (15  percent  of  those  passing)  with  the  addition  of  extra  credit  for 
veterans  preference  and  OKT  scores  compared  to  69  (2  percent  of  those 
passing)  before  adding  extra  credit.  This  is  in  contrast  to  the  OPM  test 
where  899  (29  percent  of  those  passing)  scored  90  or  above  after  adding 
extra  credit. 


Figure  12  shows  the  distribution  of  Experimental  ATC  test  scores  after 
adding  veterans  preference  and  OKT  test  score  credit  for  men  and  women. 


FIGURE  1 2 
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Table  68  also  shows  the  percentage  of  men  and  women  in  each  of  four  score 
range  groups  based  on  raw  Experimental  ATC  test  scores  (before  veterans 
preference  or  OKT  extra  credit  is  applied)  and  compares  those  to  the  per¬ 
centage  of  men  and  women  in  each  group  after  applying  extra  credit. 

TABLE  68 


EXPEftmaffM.  ATC  TEST  SCORE  GjOjJPS 


70-79 

80- 

89 

90-100 

90-99 

100+ 

With 

Raw 

With 

Raw 

With 

Raw 

With 

Score 

Credit 

Score 

Credit 

Score 

Credit 

Credit 

CN-94  7) 

(N-1156) 

fN-69) 

(N-374) 

(H-lll) 

72% 

65% 

75% 

76% 

71% 

89% 

95% 

28% 

35% 

25% 

24% 

29% 

11% 

_ 5% 

100% 

100% 

100% 

100% 

100% 

100% 

100% 

The  overall  effect  of  combining  the  Experimental  ATC  test  score  with 
veterans  preference  and  OKT  scores  is  to  increase  the  proportion  of  women 
in  the  lower  score  group  (70-79)  and  decrease  their  representation  in  the 
higher  score  group  (90+)  in  comparison  to  use  of  the  current  0P^  test. 
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Table  69  compares  the  same  Information  by  minority  groups. 

TABLE  69 

Exp.  ATC  Test  Score  Croups 
(Pass  Saaple  N-3073) 


70-79 

00- 

>89 

9Q-  1 00 

90-99 

With 

1 00+ 

Raw 

with 

Raw 

With 

Raw 

With 

Score 

Credit 

Score 

Credit 

Score 

Credit 

Credit 

(N-2090) 

(N-1475) 

(N.918) 

(N-U27) 

(N»67) 

(N-364) 

(N-109) 

White 

84X 

811 

93Z 

92Z 

94Z 

94  Z 

93Z 

Hispanic 

5Z 

61 

4Z 

4Z 

5X 

3Z 

5Z 

Black 

Ul 

131 

3Z 

4X 

1Z 

3Z 

2Z 

100% 

1006 

100% 

100% 

100% 

100% 

100% 

Again,  the  addition  of  credit  for  veterans  preference  and  OKT  scores  has  a 
relatively  small  impact  by  racial  group  except  in  the  70-79  score  group 
where  the  effect  is  to  decrease  the  percentage  of  the  white  group  by  3  percent 
(84  to  81  percent)  and  increase  the  precentage  of  the  black  group  by  2  percent 
(1J  to  13  percent). 

The  analyses  of  the  Experimental  ATC  test  battery  previously  discussed  were 
based  on  equal  weighting  of  each  of  the  three  tests  (CSC-24,  CSC-157,  and 
MCAT)  included  in  the  experimental  test  battery. 

Multiple  regression  analysis  using  the  ATC  laboratory  problem  scores  as  the 
criterion  for  a  sample  of  1827  ATC  trainees  at  the  FAA  Academy  resulted  in 
a  correlation  (R)  value  of  .5407.  Converting  the  beta  weights  to  unit 
weights  of: 


1  x  CSC  24  Arithmetic 

2  x  CSC  157  Abstract  Reasoning 

4  x  Multiple  Controller  Aptitude 

resulted  in  a  correlation  value  (R)  of  .5354.  In  order  to  assess  the  effect 
of  using  the  weighted  test  scores,  the  1978  ATC  applicant  group  test  results 
were  further  analyzed. 

Table  70  provides  the  pass/fail  results  using  the  Weighted  Experimental 
ATC  test  (excluding  OKI)  for  men  and  women.  With  the  test  weighted,  the  percentage 
of  men  who  passed  increased  from  53  percent  (unweighted)  to  56  percent.  Women 
decreased  from  48  percent  (unweighted)  to  43  percent. 


TABLE  70 


PASS/FAIL 

,  WEIGHTED 

EXPERIMENTAL  ATC  TEST, 

BY 

SEX 

Pass  Weighted 

Fall  Weighted 

Total 

lap. 

ATC  Taat 

**p-. 

AXC  Test 

Crow 

AsDlicaati 

H 

ill 

<21 

Hen 

6191 

2368 

(56X) 

1643 

(64X) 

Hon 

1785 

768 

C63X) 

1017 

(57X) 

Total 

5976 

3116 

(52X) 

2860 

(68X) 
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Table  71  provides  the  pass/fail  rates  for  selective  racial  groups  with 
the  Weighted  Experimental  AIC  test  (also  excluding  OKI).  Again,  the  mean 
score  on  the  composite  weighted  test  was  used  as  the  "cut"  score  for  the 
applicant  group. 

TABLE  71 

PASS / FAIL,  WEIGHTED  EXPERT MENTAL  ATC  1  ES  l_  BY  RACE 


Croup 

Total 

Applicant* 

Pa**  Weighted 

Exp.  ATC  Test 

-  ill 

Fall  Weighted 
Exp.  ATC  Test 

H  ill 

White 

4067 

2698 

(665) 

1369 

(341) 

Hie  panic 

339 

136 

(415) 

203 

(591) 

Black 

14Q7 

198 

(145) 

1209 

(661' 

Tocal 

5013 

3032 

(525) 

278! 

(481) 

The  pass  rate  for  the  White  group  remained  the  same  (66  percent)  as  compared 
to  the  Unweighted  test.  The  Hispanic  group  dropped  from  42  percent  to  41 
and,  the  Black  group  dropped  from  18  percent  to  14  percent. 

Table  72  provides  the  weighted  test  score  distribution  by  racial  groups. 


TABLE  72 

WEIGHTED  TEST  SCORE  BY.  RACIAL  GROUP 

(N-2781)  ;  CW-3032) 

Fall  Weighted  Exp.  AIC  Test  - j  9 -  Pee*  Weighted  Exp.  ATC  Teet 


Below 

50 

50-59 

60-69 

1 

1 

t 

70-79 

80-89 

Above 

89 

Croup 

N  £1 

N 

ill 

N 

ill 

1 

1 

1 

N 

in 

N 

in 

N 

ill 

Tocal 

whit# 

20  (•) 

273 

(  71) 

10  j 

(271) 

1 

1 

1 

1645 

(401) 

987 

(241) 

66 

(  21) 

4067 

Blapanlc 

22  (  71) 

65 

(191) 

100 

(301) 

1 

1 

1 

1 

100 

(301) 

32 

(  91) 

4 

(  11) 

339 

■leek 

187  (131) 

522 

(371) 

500 

(361) 

1 

1 

1 

1 

167 

(121) 

32 

(  21) 

0 

(  01) 

1407 

•Less  than  1 

percent 

1 

By  weighting  the  test,  more  of  the  White  applicants  shift  to  the  60-69  and  80-89 
score  ranges  and  more  of  the  Black  applicants  shift  to  the  50-69  score  range. 

As  a  total  group,  fewer  applicants  score  below  50  with  the  weighted  test  as  com¬ 
pared  to  the  Unweighted  test  (20  percent  versus  25  percent). 
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Figure  l'  shows  the  score  distribution  for  selected  racial  groups  based  on 
the  Weighted  Experimental  Test. 


FIGURE  n 


Comparing  the  Weighted  Experimental  ATC  test  battery  to  the  present  OPM  test 
in  terms  of  pass /fail  status  of  the  applicants  shows  that:  (1)  2606  passed 

both  tests;  (2)  2431  failed  both  tests;  (3)  510  passed  the  Weighted  ATC  test 
but  failed  the  OPM  test;  and,  (4)  429  passed  the  OPM  test  but  failed  the  ATC 
Weighted  Test  battery.  Use  of  these  two  different  test  affect  the  pass/fail 
status  of  939  (16  percent)  of  the  applicants.  With  the  Weighted  Experimental 
ATC  test,  510  different  applicants  (16  percent  of  those  who  passed)  would  be 
eligible  for  appointment  consideration  that  would  not  be  eligible  based  on 
the  current  OPM  test. 

In  comparing  pass /fail  results  of  the  Unweighted  and  Weighted  Experimental 
ATC  Tests,  8  percent  of  the  total  applicants  were  affected;  219  failed  the 
Unweighted  Test  but  passed  the  Weighted  Test  and  267  passed  the  Unweighted 
but  failed  the  Weighted  Test. 
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Table  7'.i  compares  the  distribution  of  the  ATC  applicants  and  those  who 
passed  the  OPM,  Unweighted  and  Weighted  ATC  test  with  the  Civilian  Labor 


Force  (CLF) , 


TABLE  7  3 

COMPARAT  l  VK  1)1  SIR  1  HI"!  1 OXS 


Croup 

Civilian 

Labor  Force 

X  of  ATC 
Applicants 

tearlcan  Indian 

.  3Z 

u 

Asian 

.  82 

a 

Black 

9.  7X 

v>x 

White 

85.  OX 

68* 

Hispanic 

4.2X 

6* 

100. OX 

100* 

X  of 

OPM 

Pass  Group 

X  of 

Exp.  ATC  Test 

Pass  Croup 

X  of  Weighted 
Exp.  ATC  Teat 
Pass  Croup 

1.0X 

1.2X 

1.1Z 

1.1X 

1.3X 

1.01 

8.9X 

7.6X 

6.4X 

84. 8X 

85. 5X 

87.  IX 

4. 2X 

4.4: 

4. 42 

100. OX 

100. ox 

100.01 

Figure  14  provides  the  distribution  of  weighted  test  scores  for  the  total 
applicant  group  in  comparison  to  the  OPM  test  score  distribution.  In  both 
cases,  the  test  scores  exclude  extra  credit  for  veterans  preference  or  OKT. 


FIGURE  14 

TOTAL  ATC  APPLICANT  tatOOP 
(N-6000) 


TEST  SCOBS  Put RtBuTtOAf 
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Table  74  shows  the  percentage  of  men  and  women  in  each  of  four  score 
range  groups  based  on  Weighted  ATC  test  scores  before  applying  veterans 
preference  or  OKT  credit  and  compares  these  to  the  percentage  after 
applying  extra  credit.  Table  7  5  provides  the  same  information  by 
racial  group. 


TABI.i:  74 

EXPERIHmAL  MEICKTED  ATC  TEST  SCORE  CROITS 


70-79 

80-89 

90-100 

90-99 

100+ 

Raw 

With 

Row 

With 

Raw 

With 

With 

Score 

Credit 

Score 

Credit 

Score 

Credit 

Credit 

(N-1968) 

(N-U26) 

(N-1076) 

CN-L 162) 

CN-72) 

(N-402) 

(N-126) 

Men 

742 

672 

772 

782 

802 

922 

962 

Women 

262 

332 

232 

222 

122 

82 

42 

1002 

1002 

1002 

1002 

1002 

1002 

1002 

TABLE 

7  5 

EXPERIMENTAl  WEIGHTED  ATC  TEST  SCORE  GROUPS 


70-79 

80-89 

90-100 

90-99 

100+ 

Raw 

Scora 

(N-1912) 

With 

Credit 

Raw 

Score 

(N-1050) 

With 

Credit 

(N-1136) 

Raw 

Score 

<37191 

With 

Credit 

(N-392) 

With 

Credit 

(N-123) 

Whita 

86* 

852 

94* 

93* 

94* 

932 

942 

Hispanic 

51 

62 

31 

3* 

6* 

42 

52 

Black 

_9X 

92 

31 

4* 

0* 

32 

12 

1001 

1002 

100* 

100* 

100* 

1002 

1002 

Table  76  provides  a  comparison  of  the  total  score  distribution  with  extra  credit 
and  the  pass  rates  for  the  present  OPM  ATC  test  battery  and  the  Experimental 
ATC  test  battery,  both  unweighted  (UW)  and  weighted  (W) .  These  distributions 
reflect  the  composition  of  the  OPM  competitive  register  from  which  selection  of 
applicants  would  be  made  based  on  the  "Earned  Rating"  for  each  of  the  test 
batteries  with  extra  credit  given  for  veterans  preference  and  OKT  scores. 

(See  Tables  59,  74,  and  56.) 


TABLE  76 

EXTRA  CREDIT  EFFECT  ON  PASS  RATES 


Tut 

Bsctary 
la  mad  Rat  ins 

70-79 

80-89 

B 

m 

H 

in 

OPM  last 

1076 

(35X) 

1086 

(36Z) 

Up.  ATC  Taat  (CW) 

1323 

(4  «X) 

1156 

(36Z) 

Up.  ATC  Taat  (V) 

1426 

(46*) 

1162 

(37*) 

90-99 

10(H 

Total 

Pace 

Total 

Applicants 

B  i*l 

b  sii 

M 

ill 

N 

644 

(21*) 

229  (•*) 

3033 

(51*) 

5976 

374 

(12*) 

111  (4t) 

3164 

(53*) 

5976 

402 

(13*) 

126  (4X) 

3116 

(32X) 

3976 

97 
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Table  7  7  provides  the  descriptive  statistics  for  the  applicant  group  on 
each  of  the  three  composite  test  batteries  by  men  and  women  and  racial 
groups . 

TAIil.K  77 

DESCK  1 1>T  1  VK  S TAT  1  ST !  CS-COMl'OS  1  TE  TEST  BATTER  I  ES 


OPM 

Test 


Group 

Mean 

SD 

Men  (N=4191) 

210.0 

46.0 

Women  (N=1785) 

199.8 

49.3 

White  (N=4067) 

220.1 

39.8 

Hispanic  (N=339) 

191.7 

49.7 

Black  (N=1407) 

165.0 

45.8 

Asian  (N=57) 

210.7 

48.3 

Amer.  Indian  (N=61) 

204.5 

41.6 

Total  Group 

207.1 

47.2 

"Pass  Score" 
Transmuted  Score 

210.0 

70.0 

Unweighted  Weighted 


Exp.  ATC 

Test 

Exp.  ATC 

Test 

Mean 

SD 

Mean 

SD 

101.9 

22.6 

227.3 

55.5 

98.3 

22.7 

210.7 

56.1 

107.5 

19.0 

241.5 

46.7 

93.9 

23.4 

206.5 

56.1 

79.6 

21.0 

170.5 

47.2 

103.8 

21.7 

235.8 

50.8 

99.2 

20.5 

222.9 

49.6 

100.5 

22.7 

222.3 

56.2 

100.47. 

70.0* 

- 

222.21 

70.0* 

Table  78  provides  the  intercorrelations  for  the  three  test  batteries. 

TABLE  78 

INTEKCORRELAT lONS-COMPOsTfE  TEST  BATTER  I ES 

Present  Exp.  Exp. 

OPM  ATC  Test  ATC  Test 

Test  Unweighted  Weighted 


OPM  1.00  .92  .88 
ATC  (Unweighted)  1.00  .97 
ATC  (Weighted)  1 • 00 


Conclusions.  Given  the  high  attrition  rate  of  ATC  trainees  and  the  objective 
of  developing  improved  methods  of  selecting  applicants  for  the  ATC  occupation, 
attention  is  focused  on  the  selection  tests  used  in  the  hiring  process.  The 
results  of  using  the  Experimental  ATC  test  battery  which  was  derived  from  the 
research  studies  previously  discussed  are  not  unexpected. 


The  number  of  applicants  who  score  high  on  the  Experimental  ATC  test  (85 
and  above)  is  much  smaller  than  on  the  current  OPM  test.  Based  on  the 
validity  studies  which  have  been  conducted,  those  applicants  who  score  high 
on  the  Experimental  ATC  test  can  be  expected  to  have  the  highest  probability 
of  success  in  the  developmental  training  for  the  ATC  occupation. 

Granting  extra  credit  for  military  service  is  required  by  law.  This  does 
impact  on  the  competitive  appointment  consideration  for  women  but  does  not 
appear  to  differentially  affect  individual  minority  groups.  Aviation-related 
experience  and  knowledge  which  is  measured  by  the  Occupational  Knowledge  test 
has  been  shown  to  be  a  positive  predictor  of  success  in  ATC  training  in  the 
research  studies  previously  discussed.  Granting  extra  credit  for  the  demon¬ 
stration  of  this  knowledge  improves  the  competitive  position  of  selected  appli¬ 
cants  but  this  is  consistent  with  the  objective  of  developing  improved  selection 
procedures  in  order  to  hire  those  applicants  who  demonstrate  aptitude  for  air 
traffic  control  work  and  whose  potential  for  success  is  greatest. 
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STUDY  OF  NEW  APPOINTEES  TO  THE  ATC  OCCUPATION  -  1978 


Objective.  Research  studies  discussed  up  to  this  point  were  directed  to: 

1)  Identifying  an  improved  ATC  test  battery; 

2)  Development  of  parallel  test  forms  for  those  tests  which 
consistently  produced  significant  correlations  with  the 
various  criterion  measures  of  ATC  success;  and 

3)  Evaluation  of  experimental  and  OPM  tests  in  relation  to 
applicants  for  ATC  work. 

Results  of  these  studies  supported  the  conclusion  that  a  new  test  battery 
comprised  of  the  Multiplex  Controller  Aptitude  Test  (MCAT) ,  Abstract 
Reasoning  (CSC-157) ,  and  Arithmetic  Reasoning  (CSC-24)  provided  consistently 
high  correlation  with  criterion  measures.  It  was  also  shown  that  the  ATC 
Occupational  Knowledge  Test  provided  a  better  measure  of  aviation-related 
knowledge  than  the  present  Rating  Guide  used  to  evaluate  experience.  The 
Occupational  Knowledge  Test  (OKT)  also  provides  significant  correlations 
with  success  in  the  ATC  occupation  which  added  to  the  validity  obtained  from 
the  test  battery  alone. 

With  the  development  of  parallel  forms  of  MCAT  and  OKT,  the  next  step  was  to 
validate  the  proposed  test  battery  with  a  new  group  of  ATC  trainees  attending 
the  FAA  Initial  Qualification  ATC  Terminal  and  En  route  training  program.  Since 
this  study  has  not  been  published  elswehere,  this  section  will  discuss  the  com¬ 
plete  methodology  and  findings. 


Sample  Description.  The  sample  of  trainees  used  for  this  validation  analysis 
consisted  of  953  new  trainees  attending  the  FAA  Academy  during  the  perioo 
June  1978  through  December  1978.  As  in  previous  studies,  tests  were  admin¬ 
istered  on  a  voluntary  basis  on  the  first  day  of  attendance  at  the  Academy. 

The  composition  of  the  sample  with  respect  to  sex  and  racial  groups  is  provided 
in  Table  79. 


Men 

Women 

Unidentified 
Tot  a  1 


TABLE  79 

SEX  AX1)  RACE  -  1978  SAMI’ EE 


800  (85 . 07.) 
141  (15.0%) 
12 

95  1 


Or ient al 

10 

(1.  J  ) 

11  i  span  i  c 

2  1 

(2.4) 

11 1  auk 

81 

(8.  5  ) 

Other 

8  59 

(88.0  ) 

Tot  a  1 

9  5'  1 

100 


Table  80  compares  the  composition  of  this  sample  with  the  1978  AT C  appli¬ 
cant  group  who  passed  the  OPM  test  and  those  who  passed  the  0PM  test  with 
total  scores  (earned  ratings)  above  89. 


TABLE  80 


197ft  SAMPLE  VFRSrS  1978  ATC  APPLICANTS 


1978 

ATC 

Passed 

Passed 

OPM 

1978  ATC 

Group 

Applicants 

OPM 

Test 

Test  90+  Score 

Trainee  Sample 

Men 

4191 

(  7 0%) 

2236 

(  74%) 

722 

(82.6%) 

800 

(  85%) 

Women 

1785 

(  30%) 

799 

(  26%) 

151 

(17.2%) 

141 

(  15%) 

5976 

(100%) 

3035 

(100%) 

873 

(100%) 

941 

(100%) 

Amer .  Ind . 

61 

(1.0%) 

31 

(1.1%) 

6 

(  .7%) 

(  “  ) 

Asian 

57 

(1.0%) 

35 

(1.2%) 

15 

(1.7%) 

10 

(1.1%) 

Hispanic 

339 

(5.7%) 

128 

(4.2%) 

31 

(3. 6%) 

23 

(2.4%) 

Black 

1407 

(23.7%) 

264 

(8.8%) 

26 

( 3.  0%) 

81 

(8.5%) 

White 

4067 

(68.6%) 

2556 

(84. 8%) 

791 

(91.0%) 

839 

(88.0%) 

5931 

(100.0%) 

3014 

(100.0%) 

869 

(100.0%) 

953 

(100.0%) 

The  representation  of  minorities  in  the  ATC  trainee  sample  is  somewhat 
greater  than  would  be  expected  if  all  trainees  had  been  selected  through 
competitive  appointment  processes  from  the  OPM  register.  Because  of  the 
large  number  of  applicants  in  relation  to  the  relative  small  number  of 
vacancies,  competitive  selections  of  ATC  trainees  are  normally  made  in 
score  ranges  above  90.  As  shown  in  the  analysis  of  the  OPM  test  scores 
for  the  1978  ATC  applicants,  the  White  group  comprised  about  91  percent  of 
all  applicants  who  scored  90  and  above  on  the  OPM  test  battery  after  in¬ 
clusion  of  credit  for  Veterans  Preference  and  aviation-related  knowledge/ 
experience.  However,  in  this  group  of  over  900  ATC  trainees,  the  White 
group  is  88  percent  of  the  total. 

Because  of  the  difficulties  in  hiring  women  and  minorities  through  com¬ 
petitive  appointment  processes  from  OPM  registers,  FAA  established  an 
alternate  recruitment  program  in  1968  which  provides  for  non-competitive 
appointment  of  individuals  who  are  already  federal  employees  with  status 
in  the  career  Civil  Service  system.  These  individuals  are  required  to 
pass  the  same  ATC  test  battery  with  a  score  of  70  or  more  as  the  appli¬ 
cants  appointed  from  the  OPM  register.  However,  having  passed  the  test, 
they  can  be  hired  at  the  GS-5  level  non-competitively .  This  alternative 
recruitment  program  is  identified  as  the  ATC  Predevelopmental  Program. 
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Approximately  200  ATC  trainees  are  hired  each  year  through  this  alternate 
recruitment  program.  This  represents  about  10  percent  of  annual  new  ATC 
hires.  These  employees  receive  a  year  of  predevelopmental  training  in 
air  traffic  control  related  subjects  including  17  weeks  of  formal  class¬ 
room  instruction  at  the  Predevelopmental  Training  Center  at  the  University 
of  Oklahoma  followed  by  training  assignments  at  FAA  terminal,  en  route 
centers,  and  flight  service  station  field  facilities.  Upon  completion  of 
predevelopmental  training,  these  employees  are  promoted  to  GS-7  (the 
normal  entry  grade  for  ATC  trainees)  and  assigned  to  the  FAA  Academy  for 
Initial  ATC  Qualification  Training. 

In  order  to  further  determine  the  composition  of  the  1978  ATC  trainee  group, 
Table  81  identifies  the  number  of  predevelopmental  employees  included  in 
the  sample  by  sex  and  racial  group. 


TABLE  81 

PREDEVELOPMENTAL  EMPLOYEES  BY  SEX  AND  RACE 


Group 

Men 

Women 


Aslan 

Hispanic 

Black 

Other 


Non-Predevelopmental  Predevelopmental 

iCompetitive  Appt.)  (Non-Competitive  Appt.)  Total 


N 

%  of 
Total 

%  of 
Group 

N 

t  of 
Total 

%  of 
Group 

N 

2 

739 

(92.42) 

(  88.92) 

61 

(  7.62) 

(  55.52) 

800 

(1002) 

92 

831 

(65.22) 

(  11.12) 

49 

(34.82) 

(  44.52) 

141 

(1002) 

(88.32) 

(100.02) 

no 

(11.72) 

(100.02) 

941 

(1002) 

9 

(90.02) 

(  1.12) 

i 

(10.02) 

(  1.02) 

10 

(1002) 

11 

(47.82) 

(  1.32) 

12 

(52.22) 

(  10.92) 

23 

(1002) 

26 

(32.12) 

(  3.12) 

53 

(67.92) 

(  50.02) 

81 

(1002) 

797 

843 

(95.02) 

(  94.52) 

42 

(  5.02) 

(  38.12) 

839 

(100%) 

(88.52) 

(100.02) 

110 

(11.52) 

(100/02) 

953 

(1002) 

t  is  evident  that  the  Predevelopmental  recruitment  program  provides  a 
®a avenue  for  eatry  of  women  and  minorities  into  the  GS-7  Initial 
Qualification  ATC  training  program.  About  12  percent  of  this  1978  ATC 
trainee  sample  entered  the  ATC  occupation  through  the  Predevelopmental 
Program.  Approximately  35  percent  of  the  women,  52  percent  of  the 
Hispanics,  and  68  percent  of  the  Blacks  in  this  group  of  over  900  ATC 
trainees  came  into  the  occupation  through  Predevelopmental  recruitment 
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Table  M  compares  Che  composition  of  this  group  of  ATC  trainees  with 
the  Civilian  Labor  Force  (CLF)  in  relation  to  both  the  ATC  trainees 
hired  through  competitive  and  non-competitive  OPM  procedures. 

I  A  HI  I  ^ 

CIVILIAN  LABOR  FORCI  VS.  AIL  lilKH'S  BY  CLASS 

Competitive  1978  Total  (Comp.  &  Non-Comp.) 


Group 

CLF 

ATC  Trainees 

1978  ATC  Trainees 

Men 

62% 

89% 

85% 

Women 

38% 

11% 

15% 

Amer.  Indian 

.3% 

_ _ 

Asian 

.8% 

1.1% 

i .  r% 

Hispanic 

4.2  % 

1.3% 

2.4% 

Black 

9.7% 

3.1% 

8.5% 

Other 

85.0% 

94.5% 

88.0% 

100.0% 

100.0% 

100.0% 

Predictors .  The  predictors  used  in  this  analysis  were: 

•  Multiplex  Controller  Aptitude  Test  (4o6e,  4e6o,  6e7o, 

6o7e,  734o,  and  7o4e) 

•  CSC  Test  24  -  Arithmetic  Reasoning 

•  CSC  Test  157  -  Abstract  Reasoning 

•  ATC  Occupational  Knowledge  Test 

OKT  101B,  100  items;  OKT  101C,  60  items,  OKI  102 
(Forms  A,  B,  C,  D,  E,  F,  G,  H) ,  80  items  keyed 

In  this  study,  two  of  the  parallel  forms  of  MCAT  were  administered  to  each 
student.  MCAT  1  was  the  first  form  administered;  MCAT  2  was  the  second. 

Table  8 5  provides  the  intercorrelation  of  raw  scores  for  each  half  or  part 
test  on  each  of  two  parallel  forms  of  MCAT  administered  to  617  ATC  students 
at  the  FAA  Academy. 

TABLES? 

INTERCORRELATIONS- MCAT  V ARALLEL  JF0 >  RMS 

1  2  3  4  5  6 

First  MCAT  Administered  (MCAT  1) 


1. 

First  Half,  Rights 

.58 

.89  .48 

.42 

.51 

2. 

Second  Half,  Rights 

.89  .48 

.49 

.56 

3. 

Total  Rights 

.89 

.89 

.60 

Second  MCAT  Administered (MCAT  2) 

4. 

First  Half,  Rights 

.52 

.85 

5. 

Second  Half,  Rights 

.89 

6.  Total  Rights 
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A  test-retesc  correlation  of  .60  between  comparable  two-part  35  minute 
forms  of  MCAT  was  obtained.  Test  reliability  was  .75  for  this  restricted 
group.  Administering  two  forms  of  MCAT  would  provide  an  estimated  reli¬ 
ability  of  about  .88  with  an  unrestricted  population.  Some  265  trainees 
in  the  total  sample  (953)  took  forms  101B  or  1Q1C  of  the  OKI.  For  pur¬ 
poses  of  this  analysis,  these  test  scores  were  converted  to  a  scale  common 
to  forms  102  of  the  OKT. 


The  descriptive  statistics  on  test  scores  for  this  sample  of  AT C  trainees 
is  provided  in  Table  S4. 

TA_B1.E__84 

DESCRIPTIVE  STATIS'I  ICS-NCAT  1978  SAMPLE 


MEN  WOMEN  TOTAL 


Q<> 

Main 

SO 

CH> 

Maan 

so 

(*) 

Hun 

sd 

MCAT  L 

(800) 

37.4 

6.9 

(141) 

34.1 

7.8 

(953) 

36.9 

7.1 

MCAT  2 

(800) 

42.9 

5.8 

(141) 

40.7 

6.4 

(953) 

42.6 

6.0 

CSC  24 

(515) 

46.6 

6.5 

(  67) 

47.2 

7.1 

(592) 

46.6 

6.6 

CSC  157 

(515) 

38.8 

6.2 

(  67) 

39.8 

6.0 

(592) 

38.8 

6.2 

OKI 

(800) 

64.9 

15.5 

(141) 

57.7 

15.1 

(953) 

63.9 

15.6 

0KIEKTAL 

HISPANIC 

SLACK 

WHITE 

T««t 

(N) 

M—n 

SO 

Q») 

Maan 

SB 

<&> 

Mann 

SO 

<5> 

Mum 

SB 

MCAT  1 

(  10) 

36.2 

4.7 

(  23) 

37.1 

6.6 

(  81) 

29.8 

7.9 

(639) 

37.6 

6.7 

MCAT  2 

(  10) 

43.2 

5.7 

(  23) 

42.7 

7.4 

(  81) 

36.6 

6.8 

(839) 

43.2 

5.5 

CSC  24 

(  1) 

44.0 

— 

(  7) 

48.7 

6.3 

(  39) 

44.2 

7.1 

(545) 

46.9 

6.5 

CSC  157 

(  1) 

43.0 

— 

(  7) 

38.7 

5.4 

(  39) 

35.3 

6.1 

(545) 

39.1 

6.2 

OKI 

(  10) 

61.3 

13.5 

(  23) 

59.0 

19.9 

(  81) 

66.2 

13.1 

(839) 

63.9 

15.8 

About  12  percent  of  the  953  ATC  trainees  entered  the  occupation  through  the 
Predevelopmental  Program.  It  was  of  interest  to  establish  the  extent  to 
which  mean  scores  and  standard  deviations  on  each  of  the  tests  differed 
between  the  Predevelopmental  (non-competitive  appointments)  and  non-prede- 
velopmental  (competitive  appointments)  groups.  Table  85  provides  the 
descriptive  statistics  for  each  of  these  groups  by  sex.  It  is  important 
to  recognize  that  the  scores  for  the  Predevelopmental  trainees  on  MCAT 
and  OKT  were  obtained  after  they  had  completed  a  year  of  training.  Scores 
o"  the  CSC-24  and  CSC-157  tests  were  obtained  for  both  groups  based  on  the 
current  ATC  test  battery  administered  prior  to  employment  in  the  ATC 
occupation. 

TABLE  85 

DESCRIPTIVE  STATISTICS  BY  SEX  -  1978  SAMPLE 


Mon-Pr adava lopaantal 
AlC  Trait mu 


Pradavalopaancal 
ATC  Tralnaaa 


nan 


H 

Mann 

(SO) 

KCAI  1 

(739) 

37.8 

(  6.6) 

MCAT  2 

(739) 

42.3 

(  5.4) 

CSC-24 

(495) 

46.7 

(  6.5) 

CSC-157 

(495) 

38. 9 

(  6.2) 

OCT 

(739) 

64.4 

(15.7) 

Woaan 


* 

Maan 

JSBI 

N 

(92) 

35.2 

(  7.4) 

(61) 

(92) 

41.5 

(  6.2) 

(61) 

(48) 

46.1 

(  7.1) 

(20) 

(48) 

40.8 

(  5.7) 

(20) 

(92) 

52.0 

(15.6) 

(61) 

Man _  Vommn 


Maan 

(SB) 

H 

Hum 

tSD) 

32.3 

(  8.1) 

(49) 

32.4 

(7.9) 

38.3 

(  8.1) 

(49) 

39.5 

(6.8) 

44.3 

(  6.6) 

(19) 

45.0 

(6.7) 

35.6 

(  5.7) 

(19) 

37.3 

(6.2) 

72.5 

(10.8) 

(49) 

66.1 

(9.4) 
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On  all  tests  except  OKT,  the  mean  score  of  the  Predevelopmental  group  is 
lower  than  the  non-predevelopmental  group.  On  OKT  it  is  evident  that  the 
year  of  predevelopmental  training  for  employees  who  are  hired  non-competitively 
is  quite  successful  in  providing  knowledge  on  air  traffic  rules,  regulations, 
and  procedures  which  the  OKT  was  designed  to  measure.  Both  men  and  women 
Predevelopmentals  score  significantly  higher  chan  their  non-predevelopmental 
counterparts  as  reflected  by  the  mean  scores  on  OKT  for  the  two  groups.  In 
addition,  the, standard  deviations  on  OKT  for  the  Predevelopmental  group  was 
considerably  lowes;  indicating  a  much  more  restricted  range  of  test  scores. 


On  the  two  present  CSC  tests  (157  and  24),  women  in  both  groups  score  somewhat 
higher  than  men.  This  is  consistent  with  the  pattern  derived  from  the  1976— 
1977  ATC  applicant  group  who  passed  the  OPM  AIC  test  and  the  1978  ATC  appli¬ 
cant  group. 


On  the  MCAT,  men  and  women  in  both  groups  increased  their  mean  test  scores 
on  the  second  administration  of  this  test.  However,  the  difference  in  mean 
scores  between  the  predevelopmental  and  the  non-predevelopmental  groups  re¬ 
mained  fairly  constant  for  each  of  the  tests.  Men  in  the  predevelopmental 
program  scored  between  4  and  5  points  lower  and  predevelopmental  women  between 
2  and  3  points  lower  on  each  of  the  tests  as  compared  to  non-predevelopmental 
trainees. 


Table  86  provides  the  descriptive  statistics  for  the  predevelopmental  and 
non-predevelopmental  groups  by  race. 

TABLE  86 


Non-Predevelopaental  ATC  Tralneee 


Oriental 

SSI 

Hispanic 

Black 

White 

Total 

SSI 

Mean 

(3D) 

Mean 

(SD) 

SSI 

Mean 

(SD) 

INI 

Kean 

(SD) 

SSI 

Mean 

(SD) 

MCAT 

l 

(8) 

36.4 

(  5.2) 

(11) 

41.7 

(  4.7) 

(24) 

30.8 

(  6.4) 

(778) 

37.7 

(  6.7) 

(821) 

37.5 

(  6.7) 

MCAT 

2 

(8) 

43.4 

(  5.7) 

(11) 

43.2 

(  5.3) 

(24) 

38.5 

(  5.6) 

(778) 

43.2 

(  5.4) 

(821) 

43.  1 

(  5.5) 

CSC 

24 

(1) 

44.0 

— 

(  4) 

51.8 

(  6.2) 

(15) 

42.4 

(  7.1) 

(532) 

46.9 

(  6.5) 

(552) 

46.9 

(  6.5) 

CSC 

U7 

«) 

43.0 

— 

(  4) 

42.0 

(  3.6) 

(15) 

35.4 

(  6.3) 

(532) 

39.1 

(  6.2) 

(552) 

39.0 

(  6.2) 

on 

(8) 

60.9 

(13.4) 

(11) 

46.1 

(20.3) 

(24) 

58.3 

(15.1) 

(778) 

63.5 

(16.0) 

(821) 

63.1 

(16.1) 

Pradcvalopaancal  ATC  Tralnaca 

Orlmtul 

Hispanic 

Black 

White 

Total 

SSI 

Mean 

(SD) 

SSI 

Maaa 

(SP) 

SSI 

Mean 

(SD) 

SSI 

Mean 

(SD) 

SSI 

Mean 

(SD) 

mcai 

1 

(1) 

34.0 

— 

(12) 

32.9 

(  5.2) 

(56) 

29.5 

(  8.3) 

(43) 

36.4 

(  6.9) 

(112) 

32.6 

(  8.1) 

MCAT 

2 

(I) 

37.0 

— 

(12) 

42.3 

(  9.2) 

(56) 

35.9 

(  7.1) 

(43) 

42.1 

(  6.0) 

(112) 

39.0 

(  7.5) 

CSC 

24 

- 

— 

— 

(  3) 

44.7 

(  4.5) 

(24) 

45.4 

(  7.0) 

(13) 

43.9 

(  6.8) 

(  40) 

44.8 

(  6.6) 

CSC 

137 

- 

— 

— 

(  3) 

34.3 

(  4.5) 

(24) 

35.3 

(  6.1) 

(13) 

38.2 

(  6.5) 

(  40) 

36.1 

(  6.1) 

OKT 

u> 

60.0 

— 

(12) 

69.1 

(13.6) 

(36) 

69.9 

(10.4) 

(43) 

70.1 

(10.3) 

(112) 

69.7 

(10.7) 

As  a  group,  the  Predevelopmental  trainees  scored  lower  on  all  tests  as 
compared  to  the  non-predevelopmental  trainees  except  on  OKT.  The  greatest 
difference  in  mean  scores  were  associated  with  the  MCAT  test  which  also  had 
greater  score  ranges  for  the  Predevelopmental  group.  These  differences 
were  generally  evident  within  each  of  the  racial  groups  as  well. 
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On  Che  OKI,  each  racial  group  of  Predevelopmentals  generally  scored  signi¬ 
ficantly  higher  with  a  more  restricted  range  of  scores  than  their  non-pre- 
developmental  counterparts.  However,  it  does  not  appear  that  the  academic 
ATC  knowledge  acquired  during  the  one  year  of  training  carrys  over,  in 
application,  to  the  MCAT  for  the  predevelopmental  group. 

On  the  MCAT,  there  is  a  consistent  pattern  of  "learning”  between  the  first 
and  second  administration  of  the  test  for  all  trainees  although  the  non- 
predevelopmental  trainees  generally  maintain  a  constant  difference  in  mean 
scores  on  both  tests  as  compared  to  the  predevelopmental  trainees.  This 
pattern  supports  the  use  of  a  double  length  MCAT  as  a  means  of  measuring 
this  "learning  ability"  on  the  part  of  applicants. 

Criterion.  In  this  analysis,  whether  or  not  the  student  passed  the  ATC 
Academy  training  course  or  failed/withdrew  was  used  as  the  criterion 
measure  against  which  the  tests  were  validated.  Of  the  students  who  with¬ 
drew,  those  who  did  so  for  medical  reasons  or  because  of  personal  or  family 
emergencies,  were  excluded  from  the  sample  since  their  training  was  inter¬ 
rupted  through  no  cause  of  the  student.  In  these  cases,  the  student  is 
allowed  to  reenter  training  after  the  medical  situation  or  emergency  is 
resolved . 

Essentially  all  of  the  students  who  fail  or  withdraw  do  so  in  the  laboratory 
phase  of  training  which  requires  the  trainees  to  demonstrate  and  apply  their 
knowledge  and  skill  to  air  traffic  control  problems  conducted  in  a  laboratory 
environment.  The  structure  of  the  grading  of  the  laboratory  phase  of  training 
is  provided  in  Figure  15.  Estimated  reliability  for  the  six  en  route  control 
problems  is  .79  and  .75  for  the  terminal  control  problems  for  students  attending 
the  FAA  Academy  during  the  period  June-December  1978. 

The  ATC  laboratory  training  phase  consists  of  six  operational  ATC  problems 
designed  to  replicate  the  non-radar  air  traffic  control  environment.  Trainees 
conduct  identical  control  problems  within  a  common  air  traffic  control  sector 
using  identical  equipment  and  evaluation  criteria  for  each  ATC  option.  These 
operational  problems  provide  stimulus  and  decision-making  conditions  repre¬ 
sentative  of  the  job  environment  and  require  responses  from  the  trainees  in 
terms  of  actions  that  are  both  possible  and  required  to  control  air  traffic 
on-the-job  with  a  minimum  of  restrictions.  Students  are  given  a  series  of 
practice  problems  before  each  of  the  six  graded  problems  as  part  of  the 
training  and  learning  process.  Each  graded  operational  problem  takes  about 
one  hour  and  is  scored  by  an  instructor  using  established  grading  standards 
for  defined  types  of  student  errors  or  deficiencies.  Additionally,  each 
student's  performance  is  evaluated  by  the  instructor  on  a  rating  scale  of 
40  to  100.  Each  problem  is  scored  and  evaluated  by  a  different  instructor. 

After  the  operational  problems,  a  Controller  Skills  Test  (CST)  is  administered. 
This  test  was  designed  to  measure  the  application  of  knowledge  and  skills 
taught  during  the  first  months  of  training.  Three  basic  elements  for  evalua¬ 
tion  are  distributed  within  the  test:  (1)  Application  of  aircraft  separation 
standards — students  must  respond  to  control  situations  presented  by  flig  c 
strips  and  charts;  (2)  responding  to  or  forwarding  information  received 
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which  pertains  to  coordination  of  aircraft  movement  or  information  with  other 
controllers;  and  (3)  other  ATC  control  items  such  as  board  management,  timeli¬ 
ness  of  actions  and  phraseology.  The  CST  is  a  50-item  multiple  choice  test 
with  an  administration  time  of  one  hour.  The  correlation  between  the  average 
score  on  the  six  operational  problems  and  the  CST  for  this  group  of  trainees 
was  .50  for  those  in  the  terminal  option  (N"454)  and  a  correlation  of  .54  for 
enroute  students  (N-473). 

During  the  academic  training  of  the  ATC  Laboratory  Phase,  a  block  test  is 
administered  for  each  subject.  A  comprehensive  knowledge  test  is  also  admin¬ 
istered  to  measure  the  degree  to  which  students  have  learned  the  academic 
portions  of  ATC  subjects  taught  during  laboratory  training.  This  comprehensive 
phase  test  is  administered  prior  to  giving  any  operational  laboratory  control 
problems.  Figure  9  provides  a  breakdown  of  the  ATC  laboratory  training  and 
the  basis  upon  which  a  student’s  total  score  for  this  phase  of  training  is 
derived.  Students  must  complete  this  phase  of  training  with  a  total  score  of 
70  or  more  to  pass  the  FAA  Academy  and  continue  on-the-job  training  at 
assigned  ATC  facilities. 

Analytical  Methodology.  The  first  step  in  the  analysis  was  the  calculation  of 
descriptive  statistics  on  the  ATC  trainee  sample  using  those  students  who  had 
test  and  criterion  data.  It  should  be  noted  that  the  sample  size  was  affected 
by  the  lack  of  complete  information  on  all  variables,  particularly  on  CSC  tests 
24  and  157  where  scores  could  not  be  retrieved  from  FAA  or  0PM  records. 

Multiple  regression  analysis  was  carried  out  using  the  experimental  test  battery 
with  and  without  OKT  to  determine  the  multiple  correlation  of  the  test  battery 
with  "pass"  or  "fail"  status  of  the  trainees.  Separate  regression  analyses 
were  conducted  for  the  total  group  and  for  subgroups!  men,  women  and  racial 
groups  where  subsample  sizes  warranted. 

The  selection  utility  of  the  test  battery  was  examined  to  assess  the  impact 
on  training  costs;  impact  on  the  sample  of  ATC  trainees  in  terms  of  those  who 
would  (or  would  not)  have  been  hired  if  the  test  battery  had  been  used  for 
selection  decisions;  and,  an  analysis  of  the  pass/fail  status  of  those  who 
would  have  been  either  eligible  or  Ineligible  for  hiring  based  on  tne  test. 
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FIGURE  1  5 


COMPONENTS  AND  WEIGHTS  USED  IN  COMPUTING  THE 
FAA  ACADEMY  A  1C  LABORATORY  TRAINING  TO  I AL  SCORE 
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6.50% 


2 . 60% 
3.90/ 
6.50% 


2.60% 

3.90% 

6.50% 


2.60% 

3.90' 

6.50' 


1 .  30% 
1.95% 
3.25" 


1  .  30" 
1.95% 
3.25% 


Controller  Skills  Test  25.00% 


Comprehensive  Phase  Test  8.00% 


_ Block  Average  2,00% _ _ _ _ _ 

*The  lab  average  constitutes  65%  of  the  total  score  for  this  phase  of 
ATC  training. 

**0n  each  lab  problem  the  instructor  gives  a  performance  rating  for  that 
problem  that  is  averaged  with  the  student's  problem  performance. 

Fince  the  ratinp  is  not  allowed  to  be  below  40.  essentially  the  student 
is  given  a  certain  amount  of  extra  credit  in  the  computation  of  the 
problem  average.  Each  of  the  six  problems  is  graded  by  a 
different  instructor. 
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Results.  The  following  discusses  the  analysis  and  findings  in  terms  of 
the  ATC  sample  used,  the  descriptive  statistics,  multiple  regression,  and 
utility  of  the  tests. 

ATC  Sample  Used.  As  pointed  out  previously,  the  sample  of  953  ATC  trainees 
was  affected  by  the  lack  of  complete  information  on  all  variables.  As  a 
consequence,  the  total  sample  was  not  available  for  some  of  the  analyses, 
particularly  the  multiple  regression.  Table  87  provides  a  distribution  of 
the  total  ATC  trainee  sample  (953)  showing  the  number  of  trainees  who  had 
data  available  for  each  of  the  variables. 

TAKl.r  87 

AYA 1 1.ABI.i:  DATA  -  I ‘>78  ATC  TRAINEE  SAMI’ 1 .1. 


Variables 

s*‘ 

Total 


Status  As 

MCAT  1 

Predev . 

MCA!  2 

CSC  24 

Pass/Fall 

NonPredev. 

OKI 

CSC  157 

Criterion 

788 

800 

515 

790 

133 

141 

67 

137 

921 

941 

582 

927 

Race 


Black 
Hispanic 
|  Aslan 
Other 
~  Total 


80 

23 

9 

821 

933 


81 

39 

23 

7 

10 

1 

839 

545 

953 

592 

79 

23 

9 

821 

932 


Availability  of  test  scores  on  CSC  24  and  157  had  the  most  impact  on  the 
ATC  sample  which  could  be  used  in  some  cf  the  analyses.  Table  88  provides 
a  comparison  of  the  means  and  standard  deviations  for  the  group  of  trainees 
with  CSC  24  and  157  scores,  the  group  without,  and  the  scores  on  those  tests 
connnon  to  both  groups. 

In  addition  to  common  test  data,  0PM  Transmuted  scores  and  Earned  Ratings 
were  available  for  many  of  these  ATC  trainees.  The  Transmuted  0PM  score 
is  the  score  the  trainee  made  on  the  present  0PM  test  battery  before  additional 
credit  for  veteran's  preference  or  aviation- re la ted  experience  is  applied. 

The  Earned  Rating  is  the  total  0PM  score  including  extra  credit  for  veteran's 
preference  and  aviation-related  experience.  Table  88  provides  the  means  and 
standard  deviations  for  these  data.  The  data  for  the  pass/fail  criterion  is 
also  provided.  These  data  were  examined  for  statistically  significant 
differences. 
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TABLE  88 

DESCRIPTIVE  STATISTICS  -  CSC  TEST  SCORES 


local 

AIC  Smjlt 


AIC  Saaplo 

With  CSC  24/15? 


AIC  Sapli 
Without  CSC  24/157 


Sit-  Toot  tot.  tffoct 

tooulto  till 


fa Tin  Tm(I 
0«c« 

SSI 

p  or 

Ho  an 

SO 

SSI 

p  or 

Ho  an 

MCAT  1 

(953) 

36.18 

(  7.12) 

(592) 

37.07 

MCAT  2 

(953) 

42.59 

(  5.96) 

(592) 

43.01 

orr 

(953) 

63.92 

(15.64) 

(592) 

63.23 

0PM  Scor*  (T) 

(704) 

17.51 

(  7.69) 

(591) 

87.21 

0PM  (U) 

(651 ) 

94.93 

(  5.66) 

(540) 

95.07 

(7) 

(939) 

.63 

(  .48) 

(585) 

.62 

9  or 

SO 

SSI 

jtoM 

so 

(  6.94) 

(361) 

36.56 

(  7.40) 

t 

m 

1.07 

4 

• 

.07 

(  5.87) 

(361) 

61.90 

(  6.05)** 

t 

• 

2.00 

4 

• 

.19 

(15.82) 

(361) 

65.04 

(15.31) 

t 

* 

-1.75 

4 

• 

.12 

(  7.62) 

(113) 

09.  OS 

(  7.88)0* 

t 

. 

-3.10 

4 

• 

.24 

(  5.43) 

(Ill) 

94.22 

(  6.61) 

t 

• 

1.44 

4 

m 

.14 

(  .49) 

(354) 

.64 

(  .48) 

s 

• 

-  .62 

h 

• 

.04 

••  (p  <  .01) 


A  t-test  was  performed  to  examine  the  significance  of  differences  on  the  tests 
and  0PM  scores.  A  z-test  was  used  to  measure  the  difference  between  the  pro¬ 
portions  for  pass/fail.  The  differences  are  not  significant  operationally. 


A  number  of  analyses  were  based  on  sex  and  race  in  addition  to  test  scores. 
Consequently,  test  scores  on  MCAT  1,  MCAT  2  and  OKT  were  examined  for  statis¬ 
tically  significant  differences  by  race  and  sex  between  the  AT C  trainees  with 
CSC  24  and  157  scores  and  those  without  scores  on  the  latter  two  tests. 

Table  89  provides  the  results.  The  "Hispanics"  and  "Asian"  groups  were  not 
used  because  of  the  small  sample  numbers. 

As  shown  in  Table  89,  group  differences  were  not  significant  for  the  "Women" 
or  "Black"  groups.  Statistically  significant  differences  were  found  for 
the  "Men"  and  "Other"  groups  where  the  sample  sizes  were  large.  However, 
the  estimated  effect  size  in  Table  89  are  all  below  the  Cohen's  criterion  (31) 
for  small. 
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TABLE  89 

MCAT  ANALYSES  BY  SEX  AND  RACE 


TOTAL  ATC 

SAMPLE 

ATC 

WITH 

SAMPLE 

CSC  24/157 

ATC  SAMPLE 

WITHOUT  CSC  24/157 

SIC.  TEST 
RESULTS 

EST.  EFFECT 
SIZE 

Group 

Taat 

ifil 

Mean 

SD 

m 

Mean 

SD 

121 

Mean 

SD 

Man 

MCAT  1 

MCAl  2 

MCAT  TOT 

OCT 

(800) 

(800) 

(800) 

(800) 

37.4 

42.9 
80.3 

64.9 

(  6.9) 

(  5-8) 

(11.25) 

(15.5) 

(515) 

(515) 

(515) 

(515) 

37.6 
43.3 
80.9 
64.  1 

(  6.6) 

(  5.7) 
(10.9) 
(15.8) 

(285) 

(285) 

(285) 

(285) 

37.0 

42.2 

79.2 
66.5 

(  7.3)* 

(  6.0)** 
(11.8)* 
(14.8) 

t  -  231 

t  -  6.11 

t  -  2.46 

t  -  -1.85 

d  -  .09 
d  •  .19 
d  -  .15 
d  •  .16 

Uomd 

MCAT  1 

MCAT  2 

MCAT  TOT 

OKT 

(i*D 

<i*D 

<i*i) 

(i*D 

34.1 

40.7 

74.8 
57.7 

(  7.8) 

(  6.*) 

(13.1) 

(15.1) 

(  67) 

(  67) 

(  67) 

(  67) 

33.3 
40.  7 
74.0 

56.3 

(  8.0) 

(  6.6) 

(13.3) 

(14.3) 

(  74) 

(  74) 

(  74) 

(  74) 

34.8 
40.6 
75.4 

58.9 

(  7.7) 

(  6.3) 
(13.0) 
(15.8) 

t  -  -  .8) 
t  -  .08 

l  -  -  .27 
t  -  -  .38 

d  -  .19 
d  -  .02 
d  •  .  1 1 
d  •  .17 

Black 

MCAT  1 

MCAT  2 

MCAT  TOT 

OKI 

(  81) 

(  81) 

(  81) 

(  81) 

29.8 

36.6 

66.4 

66.2 

(  7.9) 

(  6.8) 
(13.2) 
(13.1) 

(  39) 

(  39) 

(  39) 

(  39) 

29.  7 
36.3 
66.0 
65.6 

(  7.7) 

(  7.5) 
(13.9) 
(13.4) 

(  42) 

(  42) 

(  42) 

(  42) 

29.8 

36.9 

66.7 

66.8 

(  6.2) 

(  6.1) 
(12.8) 
(13.0) 

t  -  -  .03 

t  -  -  .26 
t  -  -  .01 

l  -  *  .14 

d  -  .01 
d  -  .09 
d  •  .  05 
d  -  .09 

Hlapauic 

MCAT  1 

MCAT  2 

MCAT  TOT 
OCT 

(  23) 

(  23) 

(  23) 

(  23) 

37.1 

42.7 

79.8 
59.0 

(  6.6) 

(  7.4) 
(12.2) 
(20.0) 

(  7) 

(  7) 

(  7) 

(  7> 

33.4 
40. 1 

73.5 
60.4 

(  6.7) 

(  5.7) 

(  9.5) 
(24.7) 

(  16) 

(  16) 

(  16) 

(  16) 

38.8 

43.8 
82.6 
58.4 

(  6.1) 

C  8.0) 
(12.5) 
(18^4) 

Asian 

MCAT  l 

MCAT  2 

MCAT  TOT 
OKT 

(  10) 

(  10) 

(  10) 

(  10) 

36.2 

43.2 
79.4 

61.3 

(  *.7) 

(  5.7) 

(  9.5) 
(13.7) 

(  1) 

(  1) 

(  1) 

(  1) 

31.0 

42.0 

73.0 

86.0 

(  ~  ) 

(  -  ) 

(  -  ) 

(  -  ) 

(  9) 

(  9) 

(  9) 

(  9) 

36.8 

43.3 

80.1 

58.6 

(  4.6) 

(  6.0) 

(  9.6) 
(11.2) 

Othac 

MCAT  1 

MCAT  2 

MCAT  TOT 
OR 

(839) 

(839) 

(839) 

(839) 

37.6 
43.2 
80.  7 
63.9 

(  6.7) 

(  5.5) 
(10.8) 
(15.8) 

(545) 

(5*5) 

(5*5) 

(545) 

37.7 

43.5 

81.2 

63.1 

(  6.6) 

(  5.4) 
(10.5) 
(15.8) 

(294) 

(294) 

(294) 

(294) 

37.4 

42.5 
79.9 
65.4 

(  6.9) 

(  5.6)** 
(11.1)» 
(15.5) 

t  •  1 . 25 

t  -  6.25 

t  *  2.17 

t  -  -1.78 

d  •  .04 
d  -  .10 

d  -  .12 
d  -  .15 

*  p(£.05) 
»*  p(£  Ql) 


The  question  as  to  whether  the  two  groups  can  be  considered  equivalent  cannot 
be  answered  definitely.  However,  the  analysis  supports  the  assumption  that 
they  are,  considering: 


No  group  differences  were  found  on  four  of  the 
measures  (Table  8F). 

No  group  differences  were  found  for  the  "Women"  or  "Black." 
groups. 


Where  group  differences  were  found,  the  estimated  effect  sizes 
were  small. 


Where  group  differences  were  statistically  significant,  they 
were  in  a  positive  direction  for  MCAT,  but  a  negative  direction 
for  OKT.  They  do  not  consistently  favor  one  group  or  the 
other. 


Table  90  shows  the  distribution  of  the  total  953  ATC  sample  in  comparison 
to  the  ATC  trainees  (592)  who  had  scores  on  all  test  variables  by  Prede- 
velopmental  and  non-predevelopmental  groups. 
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TABLE  90 

I)  I STRI BUTIONS- IN  IT  1 AL  AND  REDUCED  SAMPLES 


Total  ATC 

Sample 

Pradavalopmantai 

Non-PTadevalopmcntal 

Initial 

Reduced 

Initial 

Reduced 

Initial 

Reduced 

SeAOl* 

Sample 

Sample 

Sample 

_ S 

S“£±« _ 

S 

.  — 

Group 

N 

X 

N 

i 

N 

t 

N 

t 

H 

X 

N 

X 

Hen 

800 

(  85. OX) 

515 

(  88.51) 

61 

(  55. 5t) 

20 

(  51 . 3X) 

739 

(  88. 9X) 

495 

(  91. 2X) 

Women 

Ul 

(  15. OX) 

67 

(  11. St) 

49 

(  44. 5t) 

19 

(  48. 7t) 

92 

(  11. IX) 

48 

(  8.81) 

local 

941 

(100. OX) 

582 

(100. OX) 

110 

(100.01) 

39 

(100. OZ) 

831 

(100. 01) 

543 

(100. OX) 

Black 

81 

(  8.51) 

39 

(  6.6X) 

55 

(  50. Ot) 

24 

(  60.  OX) 

26 

(  3. It) 

13 

(  2.7X) 

Hispanic 

23 

(  2.4X) 

7 

(  1. 22) 

12 

(  10. 91) 

3 

(  7.5X) 

11 

(  l.3t> 

4 

(  .7X) 

Asian 

10 

(  ‘  I.IX) 

1 

(  .22) 

1 

(  1.01) 

- 

(  —  ) 

9 

(  1.11) 

1 

(  .2X) 

Ochar 

839 

(  88. OX) 

545 

(  92. OX) 

42 

(  38.1*) 

13 

(  **•»> 

797 

U*_J*>. 

532 

(  96. 4X) 

Total 

933 

(100. OX) 

592 

(100.02) 

no 

(100. OX) 

40 

(100. OX) 

843 

(100. OX) 

552 

(100.01) 

Descriptive  Statistics.  Of  the  total  953  ATC  trainees,  582  men  and  women 
and  592  trainees  based  on  race  had  complete  data  on  all  test  variables.  The 
next  step  was  to  compute  a  total  weighted  test  battery  score  and  descriptive 
statistics  using  MCAT  1,  MCAX  2,  CSC  157  and  CSC  24  for  the  total  group; 
those  who  passed  the  ATC  training  and  those  who  failed  or  withdrew.  In 
addition,  the  number  of  students,  means  and  standard  deviations  for  the 
Predevelopmental  trainees  (non-competitive  appointments)  and  non-predevelop- 
mental  trainees  (competitive  appointments)  are  shown  in  Table  91. 


At  this  point,  OKI  scores  were  excluded,  the  tests  used  were  weighted  as 
follows: 


2 (MCAT  1)  +  2 (MCAT  2)  +  1 (CSC  157)  +  1(CSC  24) 


Weights  were  derived  from  the  regression  analysis  which  is  discussed  later. 


TABLE_91 

DESCRIPTIVE  STATISTICS-TWO  DEVELOPMENTAL  CATEGORIES 


caoCT 

SSI 

MST 

iBl 

mi 

FAIL  — 

as 

1*0) 

mi 

TOTAL  - 

“HE! 

(SO) 

Hen 

(320) 

255.0 

(22.5) 

(199) 

2)*. 2 

(29.9) 

(5)5) 

2*7.2 

(2*. 9) 

«■ 

(  )*) 

14*.  4 

(2 2.9) 

(  )2) 

223.2 

()2. *) 

(  *7) 

235.1 

(29.9) 

Siac  It 

(  12) 

2*0.0 

05. f) 

(  2*) 

199.5 

(23. () 

(  39) 

211.5 

(29.2) 

at  i  c 

(  4) 

2**.a 

(27.)) 

(  J) 

221-0 

02.5) 

(  7) 

2)*.* 

(24.2) 

aelee 

(  1) 

113. 0 

(  —  ) 

(—  ) 

— 

<  —  ) 

(  1) 

2)3.0 

(  -  ) 

Other 

(JO) 

m.7 

(22.7) 

OH) 

2)7.0 

(27.*) 

(5*5) 

2*9.) 

(23.9) 

T«ul  Creep 

(— ) 

— 

(  —  ) 

<—  ) 

(  —  ) 

(592) 

2*3.* 

(27.*) 

■om-rUDiytumarTAL 

TUI 

mt 

aooy 

mi 

Utt 

4&1 

mi 

££ 

(IP) 

mi 

’aS 

iSi 

_ 

013) 

155.) 

(22.*) 

(17*) 

23*.  7 

(27.*) 

(*9S> 

249.5 

(25.9) 

«mm 

(  19) 

ia.i 

(22.*) 

(  21) 

23).  J 

(30.5) 

(  *9) 

2*1.* 

07.1) 

(  5) 

1)4.2 

OS.*) 

(  *) 

103.0 

09.7) 

(  15) 

21*.  3 

(22.9) 

Uepemic 

(  2> 

2*7.5 

00.*) 

(  2) 

229.0 

(  *.2) 

(  *) 

2*7.9 

(23.7) 

(  l) 

2)5.0 

(  -  ) 

(—  ) 

— 

<  -  > 

<  1) 

(  —  > 

Ocher 

039) 

23*.  9 

(22.*) 

OM) 

2)7.* 

(27.*) 

(3)2) 

2*9.) 

(25.9) 

Tetei  Cme 

<-  ) 

<  -  ) 

(-  ) 

““ 

<  ~  ) 

(552) 

247.* 

(29.1) 

nmtnummtu.  at  mam 

mi 

as 

(IP) 

mi 

m 

(90) 

mi 

'ir 

(SO) 

IMB 

(  7) 

1*2. « 

(24.1) 

(  13) 

200.* 

(27.2) 

(  20) 

2IS.1 

(33.0) 

Vee 

(  •> 

1)9.5 

(11.9) 

(  ID 

20*. 0 

(27.9) 

(  19) 

219.9 

(31.3) 

(  7) 

2*2.7 

0*4) 

(  17) 

IM.2 

(25.*) 

(  2*> 

209.7 

<31 • 5) 

Uremic 

i  is 

222.0 

(  7.1) 

(  1) 

207.0 

(  —  ) 

(  3) 

217.0 

00.0) 

{ —  ) 

(  —  ) 

(-  ) 

(  —  ) 

(—  ) 

(  7) 

2*4.7 

(».  f) 

(  •) 

217.9 

(29.  J) 

(  13) 

15*.  J 

(33.0) 

tetei  Crewe 

<-  ) 

— 

<  -  ) 

(  —  ) 

(  —  ) 

(  *0) 

219.) 
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It  should  be  pointed  out  that  the  total  of  passed  and  failed  trainees  in 
some  cases  do  not  equal  the  number  shown  for  the  total  group.  For  example, 
the  320  men  who  passed  and  the  189  who  failed  is  509  compared  to  515  shown 
for  the  total  group.  This  is  a  result  of  having  6  trainees  in  the  total  group 
for  whom  information  by  sex  was  not  available. 

The  means  and  standard  deviations  for  this  same  group  of  ATC  trainees  were 
derived, including  OKI.  The  total  weighted  test  battery  score  was  established 
as  follows:  ' 

2(MCAT  1)  +  2 (MCAT  2)  +  1 (CSC  157)  +  1 (CSC  24)  +  l (OKI) 


Since  the  OKT  is  not  intended  for  use  in  determining  if  applicants  pass  the 
ATC  test  but  rather  as  an  alternative  to  the  present  0PM  Rating  Guide  for 
establishing  extra  credit  for  aviation-related  knowledge,  it  was  given  a 
weight  of  "1"  in  deriving  the  weighted  total.  The  results  are  provided  in 
Table  92. 

TABLE  92 

EFFECT  OF  OKT  WEIGHTS  OK  VARIOUS  SAMPLES 

TOTAL  ATC  TRAINEES 


- PASS -  - FAIL -  - TOTAL 


GROUP 

SSI 

MEAN 

SD 

SSI 

MEAN 

SD 

SSI 

MEAN 

SD 

(320) 

321.7 

(25.6) 

(189) 

294.0 

(29.8) 

(515) 

311.3 

(30.3) 

UflMD 

(  3*) 

307.3 

(28.5) 

(  32) 

275.2 

(33.3) 

(  67) 

291.4 

(34.5) 

Black 

(  12) 

310.2 

(18.0) 

(  26) 

261.6 

(26.7) 

(  39) 

277.1 

(33.8) 

Hispanic 

(  A) 

312.0 

(34.3) 

(  3) 

272.3 

(11.0) 

(  7) 

295.0 

(32.8) 

Allan 

( 

1) 

319.0 

(  -  ) 

(  -  ) 

— 

(  —  ) 

(  1) 

319.0 

(  —  > 

Othar 

(345) 

320.5 

(26.3) 

(194) 

295.2 

(29.4) 

(545) 

311.3 

(30.0) 

Total  Group 

(  -  ) 

(  -  )  (  -  )  -  (  ”  ) 

NON-PREDEVILOPMENTAL  AIC  TRAINEES 

(592) 

309.0 

(31.4) 

GROUP 

SSI 

MEAN 

SD 

ifil 

MEAN 

SD 

iSi 

MEAN 

SD 

Man 

(313) 

321.6 

(25.6) 

(176) 

295.9 

(28.6) 

(495) 

312.2 

(29.4) 

Wauo 

(  26) 

308.9 

(29.0) 

(  21) 

2:>8.4 

(34.4) 

(  48) 

294.7 

(34.6) 

Black 

(  5) 

302.0 

(  8.2) 

(  9) 

259.9 

(18.2) 

(  15) 

275.0 

(24.8) 

Hispanic 

(  2) 

321.0 

(55.2) 

(  2) 

272.0 

(15.6) 

(  4) 

296.5 

(43.5) 

Allan 

( 

1) 

319.0 

(  --  ) 

(  -  ) 

— 

(  -  ) 

(  1) 

319.0 

(  —  ) 

(388) 

320.6 

(26.0) 

(188) 

295.6 

(29.4) 

(532) 

311.5 

(29.8) 

Total  Group 

(  -  ) 

(  -  )  (  "  )  ~  (  —  ) 

PRED EVELOPMENT AL  ATC  TRAINEES 

(552) 

310.5 

(30.3) 

CROUP 

SSI 

MEAN 

SD 

m 

MEAN 

SD 

SSL 

MEAN 

SD 

Man 

(  7) 

326.4 

(27.8) 

(  13) 

268.7 

(34.5) 

(  20) 

288.2 

(41.7) 

Homo 

( 

8) 

302.3 

(28.1) 

(  ID 

269.0 

(31.6) 

(  19) 

283.0 

(33.9) 

Black 

( 

7) 

316.0 

(21.3) 

(  17) 

262.9 

(33.4) 

(  24) 

278.4 

(38.9) 

Hispanic 

( 

2) 

303.0 

(12.7) 

(  1) 

273.0 

(  —  ) 

(  3) 

293.0 

(19.5) 

Aslan 

(  -  ) 

— 

(  -  ) 

(  -  ) 

— 

(  —  ) 

(  -  ) 

— 

(  —  ) 

Othar 

( 

7) 

313.4 

(39.6) 

(  6) 

285.2 

(29.0) 

(  13) 

301.5 

(37.1) 

Total  Croup 

(  -  ) 

— 

(  —  ) 

(  -  ) 

— 

(  —  ) 

(  40) 

287.0 

(38.1) 
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The  difference  in  test  performance  between  those  who  pass  and  those  who 
fail  are  consistent,  for  the  total  group  and  for  the  subgroups.  For 
example,  the  mean  score  (Table  92)  for  the  pass  group  is  consistently 
above  300  while  the  mean  scores  for  those  who  failed  is  consistently  below 
300.  Generally  there  is  a  separation  of  about  one  standard  deviation 
between  the  mean  scores  of  those  who  pass  and  fail  for  each  of  the  subgroups. 
For  the  total  ATC  sample,  the  smallest  difference  in  mean  scores  is  between 
the  "Other"  fail  group  (295.2)  and  the  "Women"  pass  group  (307.3). 

This  pattern  generally  holds  for  the  weighted  test  battery  which  excludes 
OKT  (Table  91).  The  most  evident  exceptions  are  the  mean  scores  for  the 
"Men"  and  "Other"  non-predevelopmental  "fail"  groups  compared  to  the  "Black" 
pass  group.  However,  the  number  in  the  Black  group  is  small  (5)  and  the 
mean  difference  slight,  especially  when  compared  to  difference  in  the  standard 
deviations . 

Intercorrelations  and  Multiple  Regression  Analysis.  The  next  step  was  to 
relate  test  results  on  parallel  forms  of  MCAT  and  OKT  together  with  CSC  24 
and  CSC  157  to  the  ATC  Academy  pass  or  fail/withdraw  criterion. 

Table  9 1  presents  the  intercorrelation  of  these  tests  for  the  total  ATC 
sample  and  for  each  subgroup  by  sex  and  race.  The  correlation  of  particu¬ 
lar  interest  are  those  between  the  tests  and  the  criterion.  These 
are  zero  order  validity  coefficients. 
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It  should  be  pointed  out  that  these  validity  coefficients  are  for  a  re¬ 
stricted  population.  Correction  for  restriction  in  range  was  not  a  concern 
in  this  analysis  since  the  objective  was  to  compute  regression  equations 
and  determine  the  relative  contribution  of  these  variables  for  predicting 
course  pass/fail.  However,  Table  94  provides  a  comparison  of  these  validity 
coefficients  with  the  restricted  and  unrestricted  coefficients  obtained  for 
the  prior  ATC  sample  of  1,827  trainees  who  attended  the  FAA  Academy  during 
the  period  January  1976  -  April  1978. 

TABLE  94 

VALIDITIES  -  POTENTIAL  TEST  BATTERY  VARIABLES 


ATC  Trainees* 

ATC  Trainees** 

Ju.  1976  - 

April  1978 

June  1978  -  Dec.  19 

Restricted 

Corrected 

Restricted 

r 

r 

r 

MCAT  Exp.  (Z  scores) 

.28 

.53 

MCAT  1  (Raw  acorea) 

- 

- 

.35 

MCAT  2  (Raw  acoraa) 

- 

- 

.37 

CSC  24 

.10 

.34 

.10 

CSC  157 

.07 

.40 

.18 

on 

.22 

- 

.25 

*  Criterion  •  ATC  Lab.  Average  Score  (Z  score*.  See  Tables  25  and  26) 
**  Criterion  -  Ac  ad  easy  Training  Paas/Pail  (Lab.  Training) 


The  restricted  correlations  were  next  used  in  multiple  regression  analysis. 
The  first  regression  analysis  employed  MCAT  1,  MCAT  2,  CSC  24  and  CSC  157 
as  predictor  variables.  OKI  was  excluded  since  it  is  not  Intended  as  a 
screening  test  for  applicants.  Each  of  the  four  tests  were  included  in  the 
regression  analysis  in  order  to  ascertain  its  contribution  to  the  multiple 
correlation  coefficient  (R) .  The  results  are  shown  in  Table  95. 


TABLE  95 

REGRESSION  ANALYSIS  -  FOLK  PREDICTOR  VARIABLES 


TOTAL 


- - 

ATC  SAMP 
(N-585) 

LE  - 

’--MEN— — 
(N-509) 

—WOMEN— 

(N-66) 

Teat 

R 

Beta 

R 

Beta 

R 

*£ 

Beta 

MCAT  1 

.348 

.121 

.1832 

.337 

.114 

.1800 

.375 

.140 

.1430 

MCAT  2 

.405 

.164 

.2470 

.400 

.160 

.25  5  4 

.412 

.170 

.2259 

CSC  24 

.405 

.164 

-.0109 

.400 

.160  - 

.0121 

.419 

.175 

.0355 

CSC  157 

.416 

.173 

.0997 

.406 

.165 

.0691 

.523 

.273 

.3240 

OTHER 

(H-539) 

(M-38) 

ft 

I 

»£ 

Beta 

R 

R2 

Beta 

MCAT  1 

.292 

.065 

.1538 

.565 

.319 

.4734 

MCAT  2 

.355 

.126 

.  2294 

.579 

.335 

.2016 

CSC  24 

.355 

.126  - 

.0270 

.640 

.409 

.2836 

CSC  157 

.366 

.135 

.0982 

.689 

.*75 

.2697 
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Of  the  four  tests,  CSC  24  and  CSC  157  contributed  to  the  multiple  correlation 
coefficient  for  the  "Women"  or  "Black”  groups  but  made  little  or  no  contri¬ 
bution  for  the  total  group  as  shown  below: 

Contribution  to  R 


Group 

CSC  157 

CSC  24 

Men 

.006 

_ 

Women 

.104 

.007 

Black 

.049 

.061 

Total 

.011 

— 

Other  regression  analyses  were  conducted  for  the  total  group  which  excluded 
CSC  24;  the  multiple  correlation  and  beta  weights  obtained  were  essentially 
the  same  as  those  derived  when  CSC  24  was  included. 

In  terms  of  the  total  group,  the  findings  for  CSC  24  are  consistent  with 
results  for  other  ATC  trainee  samples  and  ATC  applicant  groups.  For  the 
sample  of  1,827  ATC  students  (1976-1978)  previously  presented,  the  CSC  24 
contributed  least  to  the  multiple  correlation  and  had  the  smallest  beta 
weights  (Tables  27,  28,  and  31).  For  the  ATC  applicant  group  (1976-1977), 
CSC  24  contributed  least  to  passing  the  current  OPM  ATC  test  (Table  19). 

However,  since  CSC  24  and  CSC  157  do  contribute  to  the  prediction  of  the 
criterion  for  some  of  the  groups,  the  decision  was  to  include  them  in  the 
test  battery  for  further  analysis.  Consequently,  the  weighted  test  battery 
used  in  developing  descriptive  statistics  and  for  examination  of  the  tests 
consisted  of; 

2 (MCAT  1)  +  2 (MCAT  2)  +  1(CSC  157)  +  1(CSC  24) 


Table  96  provides  the  results  of  the  multiple  regression  analysis  for  the 
four  tests  plus  OKT.  In  this  case,  the  order  in  which  each  test  entered 
the  regression  analysis  was  determined  by  its  contribution  to  the  multiple 
correlation  coefficient.  As  shown  for  each  group,  the  tests  entered  in  a 
different  order,  especially  in  the  case  of  "Women"  and  "Blacks." 


TABLE  96 

REGRESSION  ANALYSIS-PREDICTOR  VARIABLES  AND  OKT 


TOTAL 

•ATC  SAMPLE- 


(N-585) 


Txt 

R 

r£ 

MCAT  2 

.374 

.140 

.2134 

OKI 

.429 

.184 

.2478 

MCAT  I 

.455 

.207 

.1547 

CSC  157 

.474 

.225 

.1398 

CSC  24 

.477 

.228 

.0588 

■MEN - 


(N-509) 


T«it 

l 

£ 

MCAT  2 

.369 

.136 

.2264 

OKT 

.421 

.177 

.2341 

MCAT  1 

.448 

.200 

.1539 

CSC  157 

.460 

.211 

.1109 

CSC  24 

.462 

.214 

.0530 

WOMEN- 


(N-66) 


Te»t 

R 

ii 

Itta 

CSC  157 

.403 

.162 

.2997 

on 

.531 

.282 

.3352 

MCAT  l 

.592 

.351 

.1294 

MCAT  2 

.603 

.363 

.1873 

CSC  24 

.613 

.376 

.1199 

OTSO 

(WHITE) - 


(M-539) 


Ttit 

1 

Nta 

MCAT  2 

.328 

.108 

.1924 

on 

.398 

.158 

.2583 

CSC  157 

.424 

.180 

.1397 

MCAT  1 

.438 

.192 

.1193 

CSC  24 

.440 

.194 

.0476 

- ILACT- 


T«»t 

1 

!i 

MCAT  t 

.565 

.319 

.3573 

CSC  24 

.625 

.391 

.3360 

on 

.698 

.488 

.3415 

CSC  157 

.744 

.553 

.2791 

MCAT  2 

.759 

.576 

.1974 
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Table  97  shows,  in  summary,  the  multiple  correlation  coefficient  (R)  values 
derived  by  group  and  the  variance  (R2)  accounted  for  in  the  pass  or  fail/withdraw 
criterion  based  on  the  restricted  zero-order  correlation  coefficients  (r) . 


TABLE  97 

Sl'MMAKY  YALE!  S  -  REGRESSION  ANALYSES 


Test  Test  Battery 

Battery  And  OKI 


Group 

R 

r£ 

R 

r£ 

Total  Sample 

.416 

.173 

.477 

.228 

Other 

.368 

.135 

.440 

.194 

Men 

.406 

.165 

.462 

.214 

Women 

.523 

.273 

.613 

.376 

Black 

.689 

.475 

.759 

.579 

Test  Utility.  The  purpose  of  this  analysis  was  to  examine  the  practical 
utility  of  the  new  ATC  test  battery,  relate  the  findings  to  the  information 
obtained  for  the  1978  ATC  Applicant  Group  and  estimate  the  potential  FAA 
training  investment  costs  which  could  be  saved  by  using  the  new  ATC  test 
for  selection  of  applicants. 

As  previously  noted,  the  weighted  test  scores  for  this  sample  of  585  ATC 
trainees  was  based  on: 


WTX  «  2(MCAI-1)  +  2 (MCAT-2)  +  1 (CSC  157)  +  1 (CSC  24) 

However,  for  the  1978  ATC  Applicant  Group,  only  one  parallel  form  of  MCA! 
was  administered  rather  than  the  two  which  were  given  to  this  group  of  ATC 
trainees.  In  order  to  be  able  to  relate  the  two  groups,  the  weighted  ATC 
test  scores  for  the  585  ATC  trainees  were  recomputed  using  only  the  first 
MCAI  administered  and  then  weighting  each  of  the  tests  on  the  same  basis  as 
was  used  in  the  analysis  of  the  1978  ATC  Applicant  Group.  The  weighting 
used  was: 


WT2  -  4(MCAT-1)  +  2 (CSC  157)  +  1 (CSC  24) 

The  total  raw  weighted  scores  were  then  transmuted  for  each  trainee  based  on: 


T 

s 


10.680 


RWS  -  222.27 
56.22 


•)  +  70 


as  was  shown  with  Table  77 .  Again,  a  transmuted  score  of  70  or  above  was 
equated  to  "passing"  the  new  ATC  test  for  this  ATC  trainee  group  . 
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In  order  to  assess  the  difference  between  these  two  weighting  factors  (WT^ 
and  WT  ) ,  correlation  coefficients  were  computed  between  the  two  weighted 
scores  on  the  AIC  test  battery  and  between  these  and  the  scores  on  the 
present  OPM  test  battery  which  were  also  available  for  the  ATC  trainees. 
Table  98  provides  the  descriptive  statistics  and  correlations. 

TABLE  98 

DESCRIPTIVE  STATISTICS  AND  IMTROORRELATIONS ,  ATC-OPM  TESTS 


Test 

N 

Mean 

SD 

OPM  Score  (Raw) 

591 

262.0 

23.31 

ATC  Test  (VT  Raw) 

592 

295.6 

27.57 

ATC  Test  (WT*  Raw) 

592 

272.6 

35.1 

OPM 

Score 

ATC  Score 

WT 

1 

ATC  Score 

"2 

OPM 

ATC 

Score 

Score 

-WTi 

- 

.62 

.65 

.93 

ATC 

Score 

-wt2 

* 

Since  the  weighting  used  to  examine  the  utility  of  the  new  ATC  test  battery 
(WT2)  has  a  similar  higher  correlation  coefficient  wTith  the  OPM  test  scores 
(.65  compared  to  .62),  the  results  should  tend  to  be  similar. 


The  mean  transmuted  score  on  the  present  OPM  test  battery  tor  this  group  of 
ATC  trainees  was  87.2  with  a  standard  deviation  of  7.6.  Their  mean  transmuted 
score  on  the  new  ATC  test  battery  (WT-)  was  79.6  with  a  standard  deviation  of 
6.7.  This  lower  mean  score  is  consistent  with  the  findings  derived  from  the 
1978  ATC  Applicant  Group  which  demonstrated  that  applicants  who  passed  both 
the  present  OPM  test  and  the  new  AIC  test,  generally  score  lower  on  the  ATC 
test  (see  Figure  13). 

Por  this  sample  of  585  ATC  trainees.  Table  99  compares  the  score  distribution 
for  the  present  OPM  test  battery  and  the  new  ATC  test  battery  for  the  total 
sample  and  for  the  Predevelopmental  (non-competitive  applicants)  and  Non- 
predevelopmental  (competitive  applicants)  groups.  These  are  transmuted  test 
scores  without  Veterans  Preference  or  aviation-related  knowledge/experience 
credit.  The  585  ATC  trainees  for  whom  complete  test  and  criterion  data  were  aval 
able  represent  61  percent  of  the  953  attending  the  Academy  during  June  throuah 
December  1978.  The  40  Predevelopmental  trainees  represents  36  percent  of  the 
non-competitive  hires;  the  545  Non-predevelopmental  trainees  represent  65  percent 
of  the  competitive  hires. 
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TABLE  99 

SCORE  DISTRIBUT10NS-585  TRAINEES 
total  atc  trainees 


Present 

OPM 

New 

ATC 

Score 

Test 

Test 

(WT2) 

Range 

N 

•/ 

N 

K 

95+ 

121 

20. 

2 

.  37 

90-94 

125 

2 1 .  47 

27 

4.67 

85-89 

133 

22. 67 

98 

16.  87 

80-8, 

110 

18. 87 

161 

27.67 

75-79 

63 

10.87 

147 

25.17 

70- 7  w 

33 

5.7”, 

102 

17.4- 

"Pass"  Test 

585 

1 00 .  or. 

537 

91  .87 

65-fc9 

- 

- 

39  \ 

60-6, 

- 

- 

8  } 

8.  2% 

55-59 

_ 

_ 

1  ) 

"Fail"  Test 

~ 

- 

48 

585 

100.07 

585 

100.07 

Mean 

87.2 

79.6 

sit 

7.6 

6.7 

NONPRHi.  TRAINEES 


PREDEV.  TRAINEES 


Score 

Present  OPM 
Test 

New 

Test 

ATC 

(WT2) 

Present  OPM 
Test 

Range 

N 

7 

N 

+ 

N 

7 

95+ 

114 

21 . 87 

2 

.47 

-i 

4.6 

90-9, 

122 

22 . 47 

27 

5.07 

3 

7.0 

85-89 

129 

23.  77 

97 

17.87 

4 

9.3 

80-84 

101 

18.57 

157 

28.87 

9 

23.3 

75-79 

48 

8.  87 

136 

24.97 

15 

37.2 

70-74 

26 

4.87 

90 

16.57 

7 

18.6 

"Pass"  Tests 
65-69 

545 

100.07 

509 

31  -3 

93.47 

40 

1 00 . 07 

60-64 

- 

- 

5  ( 

6.67 

_ 

55-59 

_ 

0  J 

"Fail"  Test 

~0~ 

- 

36 

~o 

_ 

545 

100.07 

545 

100.07 

40 

100.07 

New  ATC 


Test 

CUT  2) 

N 

il 

0 

0 

1 

2.57 

4 

10.07 

11 

27.57 

12 

30.0‘ 

28 

70.07 

30 . 07 

12 

40 

100.07 

Two  significant  points  are  evident  from  the  data  in  Table 


99. 


First,  48  (8.2  percent)  of  the  585  ATC  trainees,  all  of  whom  passed  the 
present  OPM  test  for  employment  eligibility,  would  not  have  passed  the  new 
ATC  test  and ' therefore  would  not  have  been  eligible  for  appointment  as  an 
air  traffic  controller  trainee.  When  the  total  group  is  further  identified 
by  competitive  hires  from  established  OPM  certificates  (Non-predevelopmental)  and 
non-competitive  hires  (Predevelopmental)  who  were  hired  through  alternate 
recruitment  procedures,  other  differences  are  apparent:  (1)  30  percent  of 
the  Predevelopmental  Group  would  not  have  been  eligible  for  appointment  in 
contrast  to  6.6  percent  of  the  Non-predevelopmental  hires  (this  6.6  percent 
compares  to  7.2  percent  of  the  1978  ATC  Applicant  Group  who  also  passed  the 
OPM  test  but  did  not  pass  the  new  weighted  (WTj)  ATC  test;  and  (2)  23  percent 
of  the  Non-predevelopmental  trainees  scored  85  or  above  on  the  new  ATC  test 
In  contrast  to  2.5  percent  of  the  Predevelopmental  group.  These  data  provide 
some  further  quantative  insight  into  the  descriptive  statistics  on  the  mean 
score  differences  between  these  two  groups  which  were  identified  in  Tables  85 , 

86,  91,  and  92.  It  should  be  pointed  out  that  the  trainees  in  this  sample  could 
not  score  lower  than  70  on  the  OKT  and  thus  had  to  have  higher  scores. 
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Second,  the  score  distribution  of  these  ATC  trainees  on  the  new  ATC  test  is 
markedly  lower  than  on  the  present  OPM  ATC  test.  About  42  percent  of  the 
total  trainee  group  scored  90  or  above  on  the  present  OPM  test  in  contrast 
to  only  5  percent  who  scored  this  high  on  the  new  ATC  test.  In  the  Non- 
predevelopmental  group,  44  percent  scored  90  or  more  on  the  OPM  test  compared 
to  5.4  percent  on  the  new  ATC  test.  Of  the  Predevelopmental  trainees,  none 
scored  above  89  on  the  new  test  but  almost  12  percent  did  on  the  OPM  test. 

These  findings  are  consistent  with  the  results  obtained  for  the  1978  ATC  appli¬ 
cants  where  passing  scores  on  the  experimental  ATC  test  were  significantly 
lower  than  on  the  OPM  test. 

Given  this  marked  difference  in  score  distribution  between  the  OPM  and  the 
new  ATC  test,  the  next  step  was  to  relate  the  scores  on  the  present  OPM  and 
new  ATC  test  to  the  pass  or  fail/withdraw  criterion.  Test  scores  and  criterion 
data  were  available  for  695  trainees  for  the  OPM  test  and  585  trainees  for 
the  new  ATC  test  (V!T2) .  The  results  are  provided  in  Table  100.  Again,  these 
scores  do  not  include  extra  credit  for  veterans  preference  or  aviation-related 
experience  or  knowledge. 


TABLE  100 


DISTRIBUTION  COMPART SONS-OPM  AND  NEW  ATC  TESTS 


ATC  TRAINEES  ATC  TRAINEES 


— 

-OPM  TEST - 

— 

- NEW 

ATC  TEST  (VT2) 

ATC  T rng 

Fall /WP 

ATC  Trn* 

Fall  M) 

Total 

Fall/WD 

R.i  t  o 

Total 

Fall 'WD 

Rato 

Benge 

N 

(X) 

N 

X 

N 

HI 

N 

% 

'« 

95+ 

145 

(20. OX) 

33 

22. 8% 

2 

(  .3X) 

0 

90-94 

148 

(21. 3X) 

47 

31. 8% 

2  7 

(  4.6X) 

-7 

18.  5X 

85-89 

156 

(22. 5X) 

62 

39.  7% 

98 

(16. 8X) 

20 

20. 4X 

80-84 

131 

(18. 8X) 

59 

45.  OX 

164 

(27. 6X) 

46 

28.  OX 

75-79 

76 

(10.  9X 

38 

50.  OX 

147 

(25. IX) 

62 

42. 2X 

70-74 

39 

(  5.6X) 

22 

56.  '.X 

102 

(1 7.4X) 

52 

51  .OX 

Below  70 

0 

- 

0 

- 

48 

(  8.2X) 

38 

79. 2X 

695 

100. OX 

261 

37. 6X 

585 

100. OX 

223 

38.  IX 

Table  lOOclearly  identifies  the  difference  between  the  present  OPM  test  and 
the  new  ATC  test  with  regard  to  the  percentage  of  trainees  who  successfully 
complete  the  ATC  training  in  each  of  the  ATC  test  score  ranges.  For  example, 
with  scores  between  90-94  almost  32  percent  of  the  OPM  test  group  fail  or 
withdraw  in  comparison  to  only  about  19  percent  of  the  group  with  test  scores 
in  this  range  on  the  new  ATC  test.  Of  particular  interest  is  the  fact  that  of 
the  48  ATC  trainees  who  scored  below  70  on  the  new  ATC  test,  79  percent  did  not 
successfully  complete  the  training. 

Table  101  provides  the  score  range  distribution  and  failures /withdrawal  rate 
for  the  585  trainees  in  relation  to  the  new  ATC  test  identified  by  Non-prede- 
velopmental  and  Predevelopmental  ATC  trainees. 
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tabu:  ioi 


fa  i  lu nr: at  rn  idrawai  ratp:  diftrieltions  by  diatilopmuotal  a-.ira'F: 


NONI’KKU.  ATC 

PREDEV 

ATC 

- TRAINEES - 

— 

- TRAINEES - 

New 

Fall  /WI) 

Fa  1  .  ' W 

ATC  Teal 

Total 

Fd 

il  1  All) 

Rail 

Tot  a  i 

Fai  1  /WI> 

Ra  ( • 

Score  Ran^-e 

N 

N 

* 

i 

N 

N 

i 

95+ 

y 

0 

01 

01 

0 

0 

. 

01 

90-9-, 

:? 

5 

2. 51 

18.51 

0 

0 

~ 

uX 

83-89 

97 

20 

io.  ix 

20.61 

1 

0 

- 

o; 

80-89 

lb? 

46 

23.11 

29.  31 

4 

0 

- 

01 

73-79 

1  3b 

57 

28.61 

91.91 

1 1 

5 

U 

o 

00 

*5 .  e,\ 

70-79 

90 

4 '3 

21 .61 

97.81 

12 

9 

37.51 

7  5  .  u; 

Below  70 

36 

28 

14. 1 X 

J77^81 

_1_2 

10 

4i  .72 

8J_.  31 

Total 

595 

199 

100.01 

36.51 

90 

29 

100.01 

60.01 

These  data  focus  attention  on  the  48  trainees  who  did  not  pass  the  new  ATC 
test  battery  and  who  would  not  have  been  eligible  for  appointment  if 
selections  had  been  based  on  the  new  test.  Of  the  48,  a  total  of  38  either 
failed  or  withdrew  from  the  training;  a  loss  rate  of  79  percent.  For  Non- 
predevelopmental  trainees  the  loss  rate  for  this  group  was  78  percent  and 
for  Predevelopmentals  it  was  83  percent. 

This  has  a  direct  and  important  impact  on  both  recruitment  for  the  air  traffic 
control  occupation  and  the  cost  of  training  for  FAA.  Had  these  48  trainees 
not  been  hired,  the  overall  loss  rate  would  be  reduced  from  38.1  percent  to 

34.4  percent  and  by  Non-predevelopmental  and  Predevelopmental  groups,  from 

36.5  percent  to  33.6  percent  and  from  60  percent  to  50  percent  respectively. 

For  each  Predevelopmental  trainee,  FAA  invests  about  $28,000  in  salary  and 
training  costs  and  about  $10,000  in  each  Non-predevelopmental  student  by 
the  time  their  FAA  Academy  ATC  training  program  is  completed.  For  each 
student  who  fails  or  withdraws  from  training,  this  investment  is  lost  and  a 
new  trainee  has  to  be  hired. 

If  the  38  trainees  had  not  been  hired  based  on  their  failure  to  obtain  a  minimum 
score  of  70  on  the  new  ATC  test,  FAA’s  loss  in  training  investment  costs  would 
have  been  reduced  by  $560,000.  Extending  this  for  2,000  new  trainees  entering 
Academy  training  on  an  annual  basis  would  result  in  a  cost  avoidance  of  about 
$1,350,000  each  year. 

This  projection  of  the  cost  avoidance  utility  of  the  new  ATC  test  battery 
addresses  only  the  benefit  obtained  if  ATC  applicants  who  scored  below  a 
minimum  of  70  on  the  new  ATC  test  battery  were  not  hired.  The  other  signi¬ 
ficant  area  where  benefits  could  be  obtained  is  to  determine  the  difference 
in  the  group  of  ATC  applicants  who  would  be  hired  under  competitive  selection 
procedures  with  the  current  0PM  test  and  the  new  ATC  test  with  allowance  of 
extra  credit  for  veterans  preference  and  aviation-related  experience  or 
knowledge. 
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In  order  Co  examine  these  differences,  the  OPM  Earned  Rating  which  is 
derived  from  the  ATC  trainee's  score  on  the  present  OPM  ATC  test  battery 
plus  additional  credit  for  veterans  preference  (0  to  10  points)  and  credit 
for  aviation-related  experience  provided  by  the  OPM  Rating  Guide  (0-15 
points) ,  was  available  for  621  of  the  843  Non-predevelopmental  ATC  trainees 
in  this  sample.  The  110  Pre-developmental  ATC  trainees  who  were  appointed 
non-competitivelv  were  not  included  in  this  analysis  since  extra  credit 
for  veterans  preference  or  aviation-related  experience  is  not  a  factor  in 
their  appointment  eligibility;  they  need  only  to  pass  the  OPM  test  battery 
with  a  score  of  70  or  above.  The  "equivalent"  of  the  OPM  Earned  Rating 
using  the  new  ATC  test  battery  (UT2)  and  the  ATC  Occupational  Knowledge  Test 
(OKT)  was  derived  for  the  Non-predevelopmental  (competitive  hires)  ATC 
trainees.  The  36  trainees  who  did  not  achieve  a  minimum  score  of  70  on  the 
new  ATC  test  were  excluded  from  the  analysis  since  they  would  not  have  been 
eligible  for  appointment.  The  "Earned  Rating"  for  these  trainees  was  based 
on  their  score  on  the  new  ATC  test  plus  additional  credit  provided  for  their 
OKT  scores  as  follows: 

OKT  Additional 

Scores  Points 

80+  15 

75-79  10 

70-74  5 

65-69  3 

The  difference  between  the  OPM  transmuted  score  and  the  OPM  Earned  Rating 
for  each  trainee  was  the  additional  credit  given  for  veterans  preference  and 
aviation  experience  based  on  the  OPM  Rating  Guide.  Since  the  veterans  pre¬ 
ference  points  were  not  separately  identified  in  the  total  of  extra  credit 
given  in  the  OPM  Earned  Rating,  it  was  not  possible  to  specifically  identify 
those  ATC  trainees  who  were  5-point  or  10-point  veterans. 

In  order  to  derive  an  "equivalent"  Earned  Rating  with  the  new  ATC  test  which 
would  reflect  extra  credit  for  veterans  preference  and  aviation-related 
knowledge  based  on  OKT  scores,  the  following  procedures  were  used: 

OPM  EARNED  RATING  (Mean  -  95.1;  N=540) 

MINUS  OPM  TRANSMUTED  SCORE _  (Mean  »  87.2;  N-591) 

-  TOTAL  OPM  EXTRA  CREDIT  (V  )  (AVG^  -  7.9  points) 

IF:  THEN: 

V  is  less  than  5  VET  points  »  0 

OKT  credit  -  0,  3,  5,  10,  15 

V.  ■  5  VET  points  ■  5 

OKT  credit  -  0,  3,  5,  10,  15 

V.  •  10,  15,  or  20  VET  points  ■  5 

OKI  credit  -  0,  3,  5,  10,  15 

V  is  more  than  20  VET  points  »  10 

OKT  credit  -  0,  3,  5,  10,  15 
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The  net  effect  of  this  procedure  would  be  a  somewhat  higher  distribution  of 
"Earned  Rating"  scores  for  the  new  ATC  test  than  would  be  the  case  if  specific 
veterans  credit  had  been  available  for  each  ATC  trainee. 

Given  these  conditions,  an  estimated  total  Earned  Rating  could  be  computed 
based  on  the  new  ATC  test  for  486  of  the  ATC  trainees  appointed  competitively 
(Non-predevelopment a 1)  from  OPM  registers.  The  48  students  who  scored  below  70 
on  the  new  ATC  Test  were  excluded  from  the  sample.  Table  102  compares  the  distri¬ 
bution  of  these  Earned  Ratings  to  the  Earned  Ratings  for  621  trainees  based  on 
the  present  OPM  ATC  test  battery  and  Rating  Guide.  The  486  trainees  for  whom 
estimated  Earned  Ratings  could  be  computed  using  the  new  ATC  test  is  a  subsamol e 
of  the  621  trainees  who  had  OPM  Earned  Ratings  available. 

TAELE  102 


NON-PREDEVEljOPMENTAL  ATC  TRAINEES 
(Competitive  Hires) 


Earned 

OPM 

Earned 

Esc. 

New  ATC 

Racing 

Ratine 

Earned  Ratine 

Score  Range 

K 

in 

N 

ill 

100+ 

205 

(33. 0Z) 

54 

(11  .  11) 

90-99 

314 

(50. 6X) 

143 

(29. 41. ) 

80-89 

99 

(15.91) 

229 

(47. IS) 

70-79 

3 

(  .51) 

60 

(12. 6)11 

Total 

621 

(100.01) 

486 

(100.01) 

About  84  percent  of  the  621  ATC  trainees  had  OPM  Earned  Ratings  of  90  or 
more.  Since  competitive  selection  from  OPM  registers  requires  hiring  those 
applicants  with  the  highest  Earned  Ratings  first,  this  distribution  of  OPM 
Earned  Ratings  would  be  expected.  However,  the  Earned  Rating  distribution 
based  on  the  new  ATC  test  for  the  486  trainees  who  are  part  of  the  sample  of 
621,  is  quite  dirferent. 

Only  41  percent  of  the  trainees  have  Earned  Ratings  of  90  or  above  and  almost 
13  percent  are  below  80  as  compared  to  less  than  1  percent  for  the  OPM  Earned 
Rating.  Consequently,  many  of  the  ATC  trainees  with  Earned  Ratings  below 
90  on  the  new  ATC  exam  would  not  have  been  hired  competitively  since  other 
applicants  with  higher  ratings  would  have  been  selected  in  their  place. 


In  terms  of  practical  utility  of  the  new  ATC  test  battery  and  0KT,  these 
differences  in  Earned  Rating  score  distribution  are  of  significance  only  if 
the  pass  or  fall /withdraw  rates  also  differ  between  the  OPM  and  new  ATC  Earned 
Rating.  These  differences  are  provided  in  Table  i<n. 


TABLE  103 


NON-PREDEVELOPMENTAL  ATC  TRAINEES 
(Coepetltlve  Hires) 

OPM  Earned  Esc.  New  ATC 

Racing  Earned  Racing 


Total 

Fail/VD  Race 

Total 

Fail/VD  Rate 

Dlff. 

Score  Range 

N 

N 

X 

N 

N 

X 

X 

100+ 

205 

54 

26. 3X 

54 

6 

11.  IX 

-15. 2X 

90-99 

314 

122 

38. 9X 

143 

33 

23.  IX 

-15. 8X 

80-89 

99 

47 

47. 5Z 

229 

86 

37. 6X 

-  9.9X 

70-79 

3 

2 

66. 6X 

60 

38 

66. 3X 

-  .  3X 

Total 

621 

225 

36. 2X 

486 

163 

33. 51 

124 


It  is  clear  from  the  data  in  Table  10'3  that  the  probability  of  successfully 
completing  the  ATC  training  program  is  greater  in  each  of  the  Earned  Rating 
score  ranges  above  79  for  the  new  ATC  test  battery. 


It  is  evident  that  an  OPM  register  based  on  the  new  ATC  Earned  Rating  would 
result  in  hiring  a  somewhat  different  group  of  ATC  trainees  than  was  in 
fact  hired  with  the  present  OPM  register.  It  is  possible  to  relate  the  data 
in  Tables  102  and  101  to  the  rosuit.s  obtained  tor  the  1978  AIC  Appl  icant  Croup 

on  both  the  current  OPM  test  and  the  new  ATC  test.  This  information  would 
provide  a  basis  for  assessing  the  differences  in  selection  of  groups  of 
applicants  and  the  potential  effect  of  these  selections  on  fail/withdraw 
rates  and  on  training  investment  losses. 


The  previous  analysis  of  the  1978  ATC  Applicant  Group  identified  (Table  7 hi 
the  Earned  Rating  score  distribution  of  applicants  who  passed  both  the 
present  OPM  test  and  the  new  weighted  ATC  test  battery  (WT_) .  If  this  group 
of  621  ATC  trainees  had  been  hired  from  the  group  of  1978  ATC  applicants  based 
on  Earned  Ratings  obtained  on  the  present  OPM  exam,  the  relationship  would  be 
as  shown  in  Table  104.  For  the  1978  ATC  Applicant  Group,  the  number  in  each 
Earned  Rating  category  in  effect  represents  the  OPM  register  from  which  applicant 
selections  would  be  made. 

TABLE  104 

PROJECTED  EARNED  "RATINGS  -  197  8  ATC  TRAINEES 


OPM  Earned 

Racing  Score 

Range 

OPM  Earned  Racing 
Qualified 

1978  ATC 

Applicancs 

n  ssi 

OPM  Earned  Racing 

1978  ATC  Trainees 
(Cavpedclve  Hires! 

N  SSI 

"Selecclon' 

Racio 

SSI 

1004- 

90-99 

80-89 

70-79 

Total 

229  (  7.5X) 

644  (  21. 2Z) 

1086  (  35. 8Z) 

1076  (  35. 5Z) 

3035  (100. 0Z) 

205 

314 

99 

3 

621 

(  33. 0Z) 

(  50. 6Z) 

(  15. 9X) 

(  .5Z) 

(100. 0Z) 

(89. 5Z) 
(48. 8X) 

(  9.1Z) 

(  .  3X) 

(20. 5X) 

The  registers  maintained  by  OPM  for  ATC  applicants  are  by  separate  employing 
jurisdictions.  Consequently,  each  of  the  eleven  FAA  regions  selects  from  its 
own  certificate  of  applicants  based  on  the  geographic  preference  identified 
by  the  applicant  and  the  Earned  Rating.  As  a  result,  one  region  (Southern, 
for  example)  may  be  able  to  fill  all  of  their  ATC  recruitment  needs  from 
applicants  who  have  Earned  Ratings  above  100.  Other  regions  (Alaska  or 
Great  Lakes,  for  example)  may  recruit  a  number  of  applicants  with  Earned 
Ratings  in  the  80’ s  or  even  70' s.  However,  Table  104  shows  that  essentially 
all  of  the  621  ATC  trainees  scored  80  or  above  and,  with  a  mean  OPM  Earned 
Rating  of  94.9  for  the  total  group,  selections  were  made  in  most  cases  from 
Che  "top"  of  the  registers  with  scores  of  90  or  more.  The  selection  ratio 
of  89.5  percent  between  the  205  1978  trainees  and  the  229  1978  applicants  who 
would  have  been  available  with  Earned  Ratings  of  100  or  more,  is  somewhat 
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higher  than  typically  experienced  by  FAA  regions.  Generally,  of  those 
applicants  offered  ATC  positions,  about  80  percent  are  selected;  the  other 
20  percent  decline  or  are  dropped  from  consideration  because  of  medical 
or  security  reasons. 

The  next  step  was  to  project  the  composition  for  a  group  of  621  ATC  trainees 
by  their  Earned  Rating  if  they  had  been  selected  from  a  competitive  register 
derived  from  the  new  weighted  ATC  test  exam  including  extra  credit  for  veterans 
preference  and  OKT  scores. 


The  number  of  1978  ATC  applicants  in  each  Earned  Rating  score  group  derived 
from  the  new  ATC  exam  was  used  as  the  "selection  register"  as  shown  in 
Table  105.  This  is  the  same  distribution  as  previously  provided  in  Table  ~h- 


A  maximum  selection  ratio  of  80  percent  in  each  Earned  Rating  group  was 
used  as  more  representative  of  actual  selection  experience.  Since  fewer 
applicants  have  Earned  Ratings  of  90  or  more  based  on  the  new  ATC  exam,  more 
selections  would  be  made  from  among  applicants  with  Earned  Ratings  in  the 
80-99  range.  Based  on  these  factors,  the  projected  distribution  of  the  621 

3  05. 


Nav  ATC 
Earned  Racing 
Score  Range 

1004- 

90-99 

80-89 

70-79 

Total 


:  new  ATC  exam 

would 

be  as 

indicated 

in  Table 

TABLE 

105 

RATING  D ■ ST R 1 

.BE  TIONS,.  1978 

ATC  AI’Pl.: 

1  CANTS 
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1978  AIC 
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»  ill 

1st.  1978 

AIC  Trainees 

Selection 

Ratio 

N 

in 

in 

126  (  A- 01) 

402  (  12.91) 
1162  (  37.31) 
1426  (  45.81) 
3116  (100.01) 

100 

322 

192 

7 

621 

(  16.11) 

(  51.91) 

(  30.91) 

(  1.11) 
(100.01) 

(SOI) 

(801) 

(171) 

Izl 

(201) 

With  the  0PM  Earned  Rating  distribution  for  the  621  ATC  trainees  and  a  pro¬ 
jected  Earned  Rating  distribution  based  on  the  new  ATC  exam,  the  fail/withdraw 
rates  provided  in  Table  88  can  be  applied  to  each  group  to  estimate  the  difference 
in  loss  rates  between  them  as  provided  in  Table  106. 

TABLE  106 

DIFFERENCES  IN  LOSS  RATE  ESTIMATES  BY  TEXT  EXAMINATION 


msmrr  hsw 
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Fall/UD 

Rav  AIC 

Eat 

ITC  EXAM - 
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N 
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N 

Rating  Scoras 

N 

in 

N 
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(  33.01) 

54 

(26.31) 

1QO+ 
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(  16.11) 

11 

(li.iD 

90-99 

314 

(  50.61) 
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(38.91) 

90-99 
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<  51.91) 

74 

(23.11) 

80-89 

99 

(  15.91) 

47 

(47.51) 
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72 
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3 

(  .31) 

2 
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70-79 

7 
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5 

1*6-61) 

Total 

621 

(100.01) 

225 

(36.21) 

Total 

621 

(100. OX) 

162 

(26.11) 
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Table  l Of  demonstrates  the  potential  utility  of  the  new  ATC  exam  in  terms 
of  an  overall  reduction  of  the  fail/withdraw  rate  of  about  10  percent  (from 
36.2  to  26.1)  for  this  sample  of  621  ATC  trainees. 

With  respect  to  FAA' s  training  investment  loss,  projecting  this  reduction  on 
an  annual  basis  for  1,800  ATC  applicants  hired  through  competitive  selection 
procedures  would  result  in  a  cost  avoidance  of  about  $1,825,000  since  signi¬ 
ficantly  more  of  the  trainees  should  successfully  complete  the  ATC  training 
program.  By  combining  this  with  the  $1,350,000  reduction  in  investment  loss 
resulting  from  not  hiring  those  applicants  who  pass  the  present  0PM  ATC  test 
but  fail  the  new  ATC  test,  the  potential  cost  avoidance  which  could  be  obtained 
is  in  the  order  of  $3,000,000  a  year. 
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UNIFORM  GUIDELINES  ON  EMPLOYEE  SELECTION  REQUIREMENTS 


The  Uniform  Guidelines  (in  jointly  established  by  the  Equal  Opportunity  Com¬ 
mission  (EEOC) ,  Department  of  Labor  (DOL) ,  Department  of  Justice  (DOJ) ,  and 
the  Office  of  Personnel  Management  (OPM),  provides  four  basic  requirements 
for  selection  .procedures  in  relation  to  equal  employment  opportunity.  Basically, 
these  requirements  are: 

•  to  determine  if  a  selection  procedure  has  an  adverse  impact  on 
employment  opportunities  of  minorities  or  women 

•  if  there  is  an  adverse  impact,  the  need  to  validate  the 
selection  procedure 

•  when  empirical  data  demonstrates  validity  of  the  selection 
procedure  (that  is,  it  is  predictive  of  or  significantly  corre¬ 
lated  with  important  elements  of  job  performance) ,  the  need  to 
examine  the  fairness  of  the  selection  procedure 

•  consideration  and  investigation  of  alternative  selection  pro¬ 
cedures.  Where  two  or  more  selection  procedures  which  are  sub¬ 
stantially  equally  valid  for  a  given  purpose,  the  procedure 
which  has  the  lesser  adverse  impact  should  be  used. 


The  various  research  studies  which  have  been  discussed  encompassed  a  wide 
range  of  selection  considerations  and  devices  both  of  a  cognitive  and  non- 
cognitive  structure.  The  analytic  methods  applied  combined  new  devices 
with  present  OPM  selection  devices  and  evaluated  their  interaction  with 
various  criteria  of  performance  to  identify  those  tests  and  procedures 
which  provided  the  highest  relationship  between  selection  procedure  scores 
and  performance  criteria  which  are  functionally  relevant  to  the  air  traffic 
control  occupation.  Based  on  this,  the  requirements  of  the  Uniform  Guidelines 
and  other  elements  of  EEO  programs  will  be  addressed  in  relation  to  the  new 
ATC  exam  and  selection  procedures  derived  from  the  research. 

Adverse  Impact.  For  enforcement  proceedings,  the  Uniform  Guidelines  state: 

"A  selection  rate  for  any  race,  sex  or  ethnic  group  which  is 
less  than  four-fifths  (4/5)  (or  eighty  percent)  of  the  rate 
for  the  group  with  the  highest  rate  will  generally  be  regarded 
by  the  Federal  enforcement  agencies  as  evidence  of  adverse  impact  .  ." 

This  definition  was  adopted  as  a  practical  "rule  of  thumb"  and  not  a  legal 
definition. 

In  the  selection  procedures  for  hiring  applicants  for  the  air  traffic  control 
occupation  addressed  by  this  research,  adverse  impact  can  occur  at  two  points 
in  the  selection  process.  The  first  point  is  establishing  basic  eligibility 
for  employment  consideration  by  successfully  passing  the  test  battery.  The 
second  point  is  the  Earned  Rating  score  (including  credit  for  veterans  preference 
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and  aviation-related  experience  or  knowledge)  which  determines  the  competitive 
ranking  of  applicants  who  pass  the  written  test. 


The  information  base  against  which  measures  of  adverse  impact  can  be  ideciified 
consist  of  the  group  of  5,976  applicants  who  took  the  present  OPM  and  experi¬ 
mental  ATC  test  battery  in  1978.  Ethnic,  race  or  sex  data  has  not  been  obtained 
by  OPM  for  other  groups  of  applicants. 


For  this  group’  of  1978  ATC  applicants,  the  relative  selection  (pass)  rates  by 
race  and  sex  on  the  present  OPM  ATC  test  battery  is  provided  in  Table  in'. 
Comparable  data  for  the  new  ATC  test  (WT2)  is  provided  in  Table  108.  Both 
tables  exclude  any  extra  credit  for  veterans  preference  or  aviation  knowledge 
since  the  purpose  is  to  evaluate  the  relative  rates  of  basic  eligibility  status 
for  the  applicants.  These  selection  rates  identify  the  proportion  of  each  sex 
and  race  group  who  passed  the  selection  test  battery  with  a  score  of  70  or 
more.  Based  on  these  rates,  the  80  percent  "rule  of  thumb"  was  applied  to  each 
group  in  relation  to  the  majority  group  selected  to  derive  the  "adverse  impact 
ratio"  which  would  occur  at  this  point  in  the  selection  process.  When  this 
ratio  is  less  than  80  percent,  the  Uniform  Guidelines  state  that  this  will 
generally  be  regarded  as  evidence  of  adverse  impact.  The  Guidelines  also 
point  out  that  where  the  ratio  is  less  than  80  percent,  it  may  not  constitute 
adverse  impact  where:  (1)  the  differences  are  based  on  small  numbers  and  are 
not  statistically  significant;  or  (2)  where  special  recruiting  or  other  programs 
cause  the  pool  of  minority  or  female  candidates  to  be  atypical  of  the  normal 
group  of  applicants.  Since  these  1978  applicants  include  a  "walk-in"  group 
resulting  from  special  recruiting  efforts  for  minorities  and  women,  the  adverse 
impact  analysis  was  done  for  the  total  group  (including  664  "walk-in"  applicants 
and  the  "Scheduled  Group"  which  would  be  representative  of  the  normal  pool  of 
applicants.  TABLE  107 
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The  data  on  the  preceeding  two  tables  show  that  including  the  "walk-in" 
applicant  group  has  relatively  little  effect  on  the  ratios  derived.  Generally, 
there  is  only  a  difference  of  one  percent  between  the  Total  Group  and  the 
Scheduled  Group  except  for  Asians  and  here  the  number  is  quite  small. 

The  present  OPM  ATC  Test  has  an  adverse  impact  on  both  the  Hispanic  and 
Black  Groups.  In  the  light  of  past  FAA  experience  in  hiring  minorities  from 
competitive  OPM  registers,  this  result  is  not  unexpected  but  Table  U'7  piv  U, 
a  quantative  measure  of  the  degree  of  adverse  impact  for  these  two  groups. 

Table  108  shows  that  the  new  ATC  test  also  has  some  adverse  impact  on  women 
applicants  in  addition  to  the  Hispanic  and  Black  Groups  and  that  the  degree 
of  adverse  impact  on  the  Black  Group  is  somewhat  higher. 

For  American  Indians,  the  addition  of  one  or  two  more  applicants  to  the  "pass 
group"  would  result  in  a  selection  ratio  of  bO  percent  or  more  on  either  the 
OPM  or  ATC  test.  Consequently,  evidence  of  adverse  impact  based  on  this  sample 
is  not  clear. 

The  analysis  was  also  completed  for  those  who  passed  the  OPM  and  Weighted  ATC 
test  to  determine  if  there  was  adverse  impact  in  each  of  three  score  range 
groups.  Candidates  who  pass  the  test  battery  are  ranked  on  competitive  OPM 
registers  based  on  their  total  score  and  those  with  the  highest  scores  selected 
first.  This  "Earned  Rating",  which  includes  additional  points  for  veterans 
preference  and  OKT  credit,  was  used  for  the  analysis.  The  results  are  provided 
in  Tables  109  (OPM  Test)  and  110  (Weighted  ATC  test). 
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ADVERSE  IMPACT  ANALYSIS 


PRESENT  OPM  TEST 

BATTERY 

WITH  VETERANS 

PREFERANCE  AND 

EXPERIENCE  CREDIT 

OPM 

OPM 

Total  1978 

Earned  Rating 
- 70  -  79 - 

Earned  Rating 

Earned  Rating 

AIC  Applicants 

1  -  "■  ■ 

Who  Passed 

OPM  Test 

N 

Ratio 

Adverse 
lapse  c 

N 

Adverse 
Ratio  Impact 

N 

Rat  lo 

Adverse 
lapse t 

Men 

Woaen 

2236 

799 

718 

358 

321 

45Z 

No 

796 

290 

36Z 

36X  No 

722 

151 

32Z 

19Z 

Yes 

3035 

1076 

1086 

873 

White 

Hiapanic 

Black 

Allan 

Aaer.  Ind . 

2556 

128 

264 

35 
_ M 

831 

50 

161 

12 

10 

33Z 

39Z 

61Z 

34Z 

32Z 

No 

No 

No 

No 

934 

47 

77 

8 

15 

36Z 

37Z  No 

29Z  • 

23Z  Yes 

48Z  No 

791 

31 

26 

15 

31Z 

24Z 

toz 

43Z 

20Z 

Yes 

No 

3014 

1064 

1081 

869 

130 


1 


I  A  HI.  I  111) 

adverse  impact  analysis 
NEW  ATC  TEST  BATTERY  lWT,> 

WITH  VETERANS  PREFERENCE  AND  EXPERIENCE  CREDIT 
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While  there  is  no  evidence  of  adverse  impact  in  the  score  ranges  of  70-89  for 
women,  both  the  0PM  test  and  the  ATC  test  have  adverse  impact  in  the  score 
range  of  90  or  more.  As  identified  earlier  (Figure  8),  a  significant  factor 
is  the  extra  credit  granted  for  veterans  preference  and,  to  a  lesser  extent, 
credit  for  aviation-related  knowledge  as  measured  by  the  Occupational  Knowledge 
Test  (OKT) .  It  is  also  evident  that  the  Weighted  ATC  test  results  in  a  greater 
adverse  impact  on  women  in  the  90  and  above  score  range  than  the  present  0PM 
test.  This  is  primarily  a  result  of  the  fact  that  raw  test  scores  for  all 
applicants  are  lower  on  the  ATC  test  than  the  0PM  test.  However,  relative  to 
men,  women  score  lower  on  the  Weighted  ATC  test.  As  shown  in  Tables  59  and 
74,  women  comprise  27  percent  of  the  group  with  raw  scores  of  90  or  more 
on  the  0PM  test  but  only  12  percent  of  this  group  on  the  Weighted  ATC  test. 

In  comparing  racial  groups  on  the  two  tests,  Tables  109  and  110  show  a  "?"  in 
some  of  the  score  ranges.  In  these  cases,  the  number  of  applicants  is  small 
and  the  addition  or  subtraction  of  one  or  two  applicants  in  the  score  range 
would  result  in  either  meeting  or  not  meeting  the  "80  percent  rule  of  thumb". 
Consequently,  evidence  of  adverse  impact  is  not  clear  based  on  this  sample  of 
1978  ATC  applicants. 

It  is  clear  that  there  is  an  adverse  impact  on  the  Black  Group  on  both  the  0PM 
and  the  ATC  test  in  the  score  range  of  90  or  more.  In  the  score  range  of  80-89 
on  the  Weighted  ATC  test,  and  possibly  on  the  0PM  test,  there  is  also  an  adverse 
impact.  Again,  this  is  primarily  a  result  of  the  fact  that  as  a  group.  Blacks 
who  passed  the  test,  score  lower  than  the  majority  group. 

In  the  case  of  the  Hispanic  Group,  there  is  possibly  adverse  Impact  in  the  score 
range  of  90  or  more  on  the  0PM  and  ATC  test  but  not  in  the  range  of  80-89  for 
the  0PM  test. 
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In  terms  of  competitive  employment  opportunities,  based  on  the  present  OPM 
ATC  test,  between  80  and  85  percent  of  the  appointments  are  generally  made 
from  among  applicants  with  Earned  Ratings  of  90  or  more.  Consequently, 
employment  prospects  of  applicants  with  scores  in  the  80-89  range  would  be 
characterized  as  "poor"  to  "fair".  Based  on  the  Weighted  ATC  test,  however, 
about  40  percent  of  the  applicants  hired  would  have  scores  of  90  or  more. 
Consequently,  employment  prospects  of  applicants  with  scores  in  the  80-89 
range  on  the  ATC  test  would  be  "fair"  to  "good"  (Table  102). 


When  the  score  range  of  80  or  more  on  the  Weighted  ATC  test  was  examined, 
there  is  an  adverse  impact  for  the  women  and  Black  Groups.  For  the  Hispanic 
and  American  Indian  Groups  the  numbers  are  small  and  the  addition  of  one 
or  two  more  applicants  in  the  score  range  would  meet  the  "80  percent" 
criterion.  The  results  are  provided  in  Table  HI- 
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This  analysis  shows  the  Weighted  ATC  test  has  an  adverse  impact  on  women  and 
some  minority  groups  with  respect  to  both  initial  eligibility  (in  passing  the 
test)  and  by  using  the  total  scores,  including  extra  credit;  for  ranking  success¬ 
ful  candidates.  In  this  situation,  the  Uniform  Guidelines  require  evidence  of 
the  validity  of  test,  the  basis  for  establishing  the  cutoff  pass  score  and  the 
>se  of  scores  for  rank  ordering  of  candidates. 

Test  Validity,  Pass  Score  and  Candidate  Ranking.  The  validity  of  the  new  ATC 
test  battery  as  well  as  individual  test  components  of  the  battery  has  been 
demonstrated  in  relation  to  a  number  of  ATC  performance  criterion  measures. 

In  summary: 

•  1977  IC  Study.  For  a  sample  of  approximately  2,100  full 

performance,  developmental  and  newly  hired  ATC  specialists, 

MCAT,  Directional  Headings,  Occupational  Knowledge  (OKI)  and 
the  Prior  Experience  Questionnaire  (PEQ)  provided  significant 
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correlations  (p  £  .01)  with  the  Aggregate  ATC  Success  criterion 
for  each  of  four  ATC  options  (FSS,  VFR,  IFR  and  ARTCC)  and  for 
all  opcions  combined  (see  Table  5).  Tn  addition,  mean  test 
scores  for  each  of  the  four  ATC  options  were  different  at  statis¬ 
tically  significant  levels  of  confidence  (p  ^  .01)  or  (p  .05) 
with  the  FSS  option  average  score  lowest  and  the  ARTCC  score 
highest  (see  Table  6). 

•  1976-1978  ATC  Study.  For  a  sample  of  1,827  ATC  trainees,  experi¬ 
mental  forms  of  MCAT  together  with  Abstract  Reasoning  (0PM  157) 
and  Arithmetic  Reasoning  (0PM  24)  provided  a  multiple  correlation 
(corrected  for  range  restriction)  of  .54  with  the  ATC  laboratory 
problem  average  score  criterion  (see  Table  28).  Cross  validation, 
using  the  weighted  test  scores  derived  from  the  multiple  regression 
provided  a  multiple  correlation  value  (R)  of  .5381  for  one  sample 
and  .5292  for  the  other  (see  table  32).  When  test  scores  on  the 
Occupational  Knowledge  Test  (OKI)  were  combined  with  the  experi¬ 
mental  test  battery  (MCAT,  0PM  157  and  0PM  24),  an  estimated 
multiple  correlation  value  of  .60  was  obtained  with  the  increase 
significant  at  the  .01  level  of  confidence. 

•  1978  ATC  Study.  For  a  new  sample  of  585  ATC  trainees,  parallel 
forms  of  MCAT  together  with  the  present  0PM  tests  157  and  24  pro¬ 
vided  a  multiple  correlation  value  (R)  of  .42  for  the  total  sample. 

The  R  values  by  sex  and  race  ranged  from  .37  to  .69  (see  Table  95). 

The  criteria  used  was  the  pass  or  fail/withdraw  status  of  the 
trainee  at  the  completion  of  ATC  Laboratory  training.  Validity 
coefficients  (r  values)  used  in  the  multiple  regression  were  not 
corrected  for  range  restriction  (see  Table  94  ).  When  test  scores 
on  parallel  forms  of  0KT  were  combined  with  the  weighted  test 
battery,  the  multiple  correlation  value  for  the  total  sample  was 
.48  with  R  values  by  sex  and  race  ranging  from  .44  to  .76  (see 
Table  96.  The  increase  in  the  variance  (R^)  accounted  for  by  the 
weighted  test  battery  with  0KT  was  significant  at  the  .01  level  of 
confidence  for  the  total  sample  and  for  each  race  group.  For 
women  the  increase  was  significant  at  the  .05  level  (see  Table  9"  ). 

The  utility  of  the  new  ATC  test  battery  (as  compared  to  the  present  0PM 
battery)  was  examined  with  respect  to  appointment  eligibility  (pass  scores) 
and  ranking  of  successful  candidates  based  on  total  scores  including  credit 
for  veterans  preference  and  ATC-related  knowledge. 

To  do  this,  the  information  available  for  the  1978  ATC  trainee  sample  was 
related  to  the  1978  ATC  applicant  group.  A  common  test  battery,  consisting 
of  parallel  forms  of  MCAT,  0PM  157  and  0PM  24  was  administered  to  both  groups. 
For  the  OKT,  a  60-item  test  (101— C)  was  administered  to  the  1978  ATC  applicant 
group  and  parallel  forms  of  OKT  102  (80  item)  administered  to  most  of  the 
1978  ATC  trainee  sample.  The  common  test  battery  was  weighted: 

4  x  MCAT  Total  Right  Scores 
2  x  0PM  157  Total  Right  Scores 
1  x  0PM  24  Total  Right  Scores 
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This  weighting  differs  somewhat  from  the  weights  derived  for  the  test  battery 
which  included  two  parallel  forms  of  MCAT  for  the  1978  ATC  trainee  group 
(see  page  114).  However,  the  correlation  coefficient  between  the  two 
weighted  ATC  tests  was  .93  (see  Table  98). 

In  the  analysis  of  the  new  ATC  test  battery,  the  passing  score  was  set  at 
the  mean  raw  score  for  the  1978  ATC  applicant  group  (222.27).  This  was  then 
equated  to  a  transmuted  passing  score  of  70  (see  Table  77).  With  this  base, 
equivalent  scores  on  the  same  weighted  ATC  test  battery  were  computed  for 
385  of  the  1978  ATC  trainees.  A  total  of  48  ATC  trainees  (8.2  percent)  failed 
to  achieve  a  score  of  70  on  the  new  ATC  test.  Of  these  48  trainees,  79  percent 
(38)  failed  the  ATC  Laboratory  training  or  withdrew  (see  TablelOO).  Some  39 
of  the  48  trainees  had  scores  in  the  65-69  range  and  9  scored  below  65  (see 
Table  99).  The  score  range  of  70-74  on  the  ATC  test  included  102  of  the  585 
trainees  (17  percent).  Of  this  group,  51  percent  (52)  failed  or  withdrew 
(see  Table  1(10).  Consequently,  use  of  the  mean  test  score  resulted  in  a  28 
percent  reduction  in  the  fail/withdraw  rates  between  the  ATC  trainees  who 
scored  in  the  70-74  range  in  contrast  to  those  who  scored  below  70  on  the 
ATC  test.  The  difference  between  a  fail/withdraw  rate  of  51  percent  in  the 
70-74  score  range  and  a  rate  of  79  percent  below  scores  of  70  clearly  supports 
the  "cutoff"  score  developed  in  the  analysis. 

The  use  of  scores  on  the  ATC  test,  with  and  without  extra  credit  for  veterans 
preference  and  ATC-related  knowledge,  as  a  basis  for  ranking  successful  ATC 
candidates  was  also  examined.  Since  extra  credit  is  a  selection  consideration 
only  for  those  candidates  who  are  appointed  from  competitive  OPM  registers, 
the  analysis  included  only  those  1978  ATC  trainees  who  were  hired  competitively 
(Non-predevelopmental)  and  who  scored  70  or  above  on  the  ATC  test.  On  this 
basis,  509  of  the  585  trainees  were  included  in  the  analysis. 

For  the  ATC  test  battery  without  extra  credit,  Table  101  (the  Non-predevelop- 
raental  ATC  Group)  shows  that  the  fail/withdraw  rate  in  each  score  range  pro¬ 
gressively  decreases  as  the  test  scores  increase.  In  the  95-100  range  there 
were  no  failures  or  withdrawals  (only  2  trainees  were  in  this  range).  In  the 
70-74  range,  47.8  percent  failed  or  withdrew.  The  largest  increase  in  the 
fail/withdraw  rate  is  between  the  75-79  range  (41.9  percent)  and  the  80-85 
range  (29.3  percent);  a  difference  of  12.6  percent  in  the  fail/withdraw 
rate.  Table  110  combines  the  ATC  test  score  ranges  and  provides  the  failure/ 
withdrawal  rates  for  this  sample  of  509  trainees. 


I  ABLE  112 


1978  ATC  THAI HIES 
ATC  TEST  SCORES 
(WITHOUT  EXTRA  CREDIT) 


Scor* 

rail /Withdraw 

Into 

» 

hll/UltUriv 

Rata 

90  ♦ 

29 

5 

17.21 

80-89 

2SA 

66 

28.01 

70-79 

Tot*) 

228 

509 

100 

171 

A4.2Z 

33761 
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These  data  clearly  show  that  ranking  ATC  candidates,  based  on  their  test 
scores  and  selecting  those  with  the  higher  scores  first,  will  result  in 
hiring  applicants  with  the  highest  probability  of  successfully  completing 
the  ATC  training  program. 

Ranking  candidates  based  on  total  "Earned  Rating"  scores  which  include  test 
scores  plus  extra  credit  for  veterans  preference  and  ATC-related  knowledge 
based  on  OKT  scores  was  examined  next.  From  the  sample  of  509  ATC  trainees 
who  were  hired  competitively,  the  equivalent  of  the  0PM  Earned  Rating  for 
the  ATC  test  could  be  computed  for  486  trainees.  Table  1  I  >  provides  the 
fail/withdraw  rates  by  Earned  Rating  score  ranges  and  compares  these  to  the 
rates  based  on  the  ATC  test  score  ranges  without  extra  credit. 

tabu:  113 

1978  ATC  TRAINEES 

(Non-Predevelopmental— Competitive  Hires) 


Score  Range 

N 

- ATC  Test 

(Without  Extra 

Fall /Withdraw 

Credit) 

Fail  Withdraw 
Rate 

N 

(With  Extra  Credit' 

Fall  Withdraw 
Fai  1  /Withdraw  Rate 

90  ♦ 

29 

5 

17.21 

197 

39 

i9.s: 

80-89 

254 

66 

26. 02 

229 

86 

3'.  &: 

70-79 

226 

100 

44. 2X 

60 

38 

63.3: 

Total 

509 

171 

33.62 

486 

163 

33.5: 

It  is  evident  from  Table  hi! that  ranking  candidates  for  selection  consideration 
on  the  basis  of  the  Earned  Rating  (which  includes  extra  credit)  also  results 
in  selecting  those  applicants  first  with  the  highest  probability  of  successfully 
completing  the  ATC  training.  It  is  also  evident  that  selection  based  on  the 
Earned  Rating  results  in  a  higher  fail/withdraw  rate  in  each  score  range  than 
if  candidates  were  hired  solely  on  the  basis  of  their  test  battery  scores  without 
granting  extra  credit. 

In  deriving  the  Earned  Rating  score  for  this  sample  of  486  ATC  trainees,  extra 
credit  for  veterans  preference  was  included  since  it  is  granted  by  law.  Extra 
credit  based  on  OKT  test  scores  was  included  based  on  the  positive  and  signi¬ 
ficant  correlations  with  ATC  performance  criterion  used  in  the  various  research 
studies. 

Consequently,  it  was  hypothesized  that  extra  credit  for  being  a  veteran, 
which  has  not  been  examined  for  validity  with  the  criterion  measures  used  in 
the  research,  does  not  predict  (or  negatively  predicts)  success  In  the  ATC 
Laboratory  training.  In  order  to  examine  this,  scores  for  the  509  ATC 
trainees  were  computed  which  excluded  any  veterans  credit  and  included  only 
the  weighted  ATC  test  battery  score  and  extra  credit  for  OKT  scores. 
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Table  I  14  provides  Che  score  distributions  for  the  group  of  509  1978  ATC 
trainees  before  and  after  the  application  of  extra  credit  for  air  traffic 
control  related  knowledge  based  on  OKT  points. 

TABLE  114 


1978  ATC 

I  RA  I NETS  -  T 

FFEC'I 

OF  EXTRA  ( 

IKED  I  1' 

(Distribution  Without  OKI  Credit) 

(Distribution 

i  vlth  OKI  Credit 

N 

2 

K 

X 

90+ 

29 

5.72 

90+ 

139 

27.32 

80-89 

254 

49.92 

80-89 

245 

•« 

OO 

i 

70-79 

226 

44.42 

70-79 

125 

24.62 

To  uil 

509 

100.02 

Total  509 

100.02 

It  is  significant  to  note  the  addition  of  OKT  points  resulted  in  moving 
110  of  the  ATC  trainees  to  a  score  range  of  90  or  more — an  increase  of 
about  22  percent.  It  also  resulted  in  reducing  the  number  of  trainees 
in  the  lowest  scoring  group  (70-79)  by  91 — from  226  to  125 — a  reduction  of 
about  20  percent. 

Based  on  the  validity  data  previously  established  for  the  OKT,  it  would  be 
expected  that  those  trainees  who  moved  to  the  higher  score  ranges  as  a 
result  of  OKT  extra  credit  would  have  lower  fail/withdraw  rates  and  that 
those  who  remained  in  the  lowest  score  range  would  have  higher  fail/withdraw 
rates.  The  comparison  of  the  fail/withdraw  rates  in  the  three  score  ranges 
for  the  509  ATC  trainees  before  and  after  applying  OKT  extra  credit  points 
is  provided  in  Table  115. 

TABLE  115 

1978  ATC  TRAINEES  FAIL/WTTHDRAW  RATES7 EFFECTS  OF  OKT  CREDIT 


- 1978  ATC  Treineea -  - 1978  Treineee 


N 

(Without  on 
7*11/ 
Withdraw 

Credit) 

F/VTD 

Rate 

N 

(With  OKI  Cradlt) 
rail/ 

Withdraw 

7/WD 

Kate 

90+ 

29 

5 

17.21 

139 

19 

13.71 

80-89 

254 

66 

26.02 

245 

78 

31.82 

70-79 

226 

100 

4V2I 

125 

74 

59^22 

Total 

509 

171 

33.62 

509 

171 

33.62 

Since  the  fail/withdraw  rate  for  the  total  group  of  509  trainees  is  a  con¬ 
stant  (33.6%),  the  rates  can  only  vary  between  the  different  score  ranges 
as  shown  in  Table  11 5-  in  order  to  assess  the  effect  of  OKT  extra  credit 
on  the  total  fail/vithdraw  rate,  one  can  assume  a  selection  made  from  among 
the  group  of  509  trainees.  For  example,  if  one  were  to  hire  300  trainees 
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from  the  509  in  rank  order  (highest  score  range  first)  with  an  estimated 
selection  ratio  of  80  percent  in  each  score  range,  the  fail/withdraw  rate 
for  the  300  hired  before  application  of  OKT  extra  credit  would  be  30  percent. 
The  fail/withdraw  rate  for  the  300  selected  after  application  of  OKT  extra 
credit  would  be  25  percent.  If  only  100  of  the  509  were  selected,  the 
fail/withdraw  rate  for  the  group  without  OKT  extra  credit  would  be  24  percent 
in  contrast  to  14  percent  for  the  group  with  OKT  credit.  These  data  support 
the  conclusion  that  ranking  applicants  for  selection  consideration  based 
on  test  scores  and  OJCT  credit  will  result  in  hiring  applicants  with  the 
highest  probability  of  successfully  completing  ATC  training. 

Table  life  shows  the  fail/withdraw  rates  for  the  509  trainees  based  on  (1)  the 
ATC  test  alone; (2)  the  ATC  test  plus  OKT  credit;  and,  (3)  the  ATC  test  plus 
credit  for  OKT  and  Veterans  Preference  points. 


TABLE  116 

1978  ATC  TRAINEES  FAIL /WITHDRAW  RATES  -EFFECT  OF  CREDITS^ 


(No 

ATC  Teat-  ■  _r" 
Extra  Credit) 

(riua 

OKT  Extra 

Credit 

Only) 

(Plua 

OKT  and 

Vet.  Pref. 

Credit ) 

Score 

X 

Fail 

F/WD 

X 

Fail 

t/wd 

X 

Fail 

F/WD 

Range 

N 

N 

Withdraw 

Rate 

N 

N  Withdraw 

Rate 

N 

N 

Withdraw 

Rate 

90  ♦ 

29 

(  62) 

5 

(17. 22) 

139 

(  272) 

19 

(13.72) 

197 

(  412) 

39 

(19.82) 

80-89 

C  501) 

66 

(26.01) 

245 

<  482) 

78 

(31.82) 

229 

(  472) 

86 

(37. 6X) 

70-79 

226 

(  941) 

100 

(44 . 21) 

125 

(  252) 

74 

(59.22) 

60 

(  122) 

38 

(63.32) 

Total 

309 

(100!) 

171 

<33. 6X> 

309 

(1002) 

171 

(33.62) 

486 

(1002) 

163 

(33.52) 

The  use  of  OKT  credit  resulted  in  moving  110  trainees  into  the  90  and  above 
score  range  compared  to  the  ATC  test  alone.  The  number  of  fail/withdraws 
increased  by  14  for  a  rate  of  13  percent  for  this  group.  When  veterans 
preference  credit  is  also  applied  for  486  of  the  509  trainees,  an  additional 
58  trainees  moved  to  the  90  and  above  score  range  and  the  number  of  fail/ 
withdraws  increased  by  20;  a  fail/withdraw  rate  of  34  percent  for  this  group  of 
58  students.  From  the  data  presented  in  Table  116,  granting  extra  credit  for 
veterans  preference  increased  the  fail/withdrawal  rates  in  the  higher  score 
ranges  and  therefore  has  a  negative  relationship  to  potential  success  in 
ATC  training. 

Test  Fairness.  When  empirical  evidence  of  validity  for  a  selection  proce¬ 
dure  such  as  presented  for  the  new  ATC  test  has  an  adverse  impact  on  a  race, 
sex  or  ethnic  group,  the  Uniform  Guidelines  require  examination  of  test 
fairness  where  technically  feasible. 

In  establishing  the  Uniform  Guidelines  on  Employee  Selection,  it  was  recog¬ 
nized  there  is  serious  debate  on  the  question  of  test  fairness.  There  are 
several  competing  definitions  of  test  or  selection  fairness,  each  of  which 
incorporates  a  different  set  of  social  values.  Reference  (3 1 )  provides  an 
analysis  of  three  incompatible  ethical  positions  in  regard  to  fair  and  un¬ 
biased  use  of  tests  together  with  differing  statistical  definitions  of  "test 
fairness"  and  their  relationship  to  specific  ethical  positions  These  three 
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ethical  definitions  are  characterized  as: 


•  Unqualified  Individualism 
o  Qualified  Individualism 

•  The  Quota  Ethic 

The  General  Accounting  Office,  in  examining  Federal  selection  tests  and  examina¬ 
tions  (33)  stated  with  respect  to  these  three  definitions: 

"The  published  literature  on  test  validity  indicates  that  most  tests 
are  either  fair  to  minority  groups  or  slightly  biased  in  their 
favor  by  the  second  definition,  (Qualified  Individualism)  which 
according  to  OPM,  is  the  only  concept  of  fairness  consistent  with 
merit  system  principles.  The  literature  also  indicates  that  by  the 
first  definition,  (Unqualified  Individualism)  tests  are  slightly 
biased  against  minority  groups,  and  if  one  subscribes  to  the  last 
definition  (Quota  Ethic)  tests  have  always  been  biased  against 
minorities. " 

The  "Qualified  Individualism"  definition  of  fairness  adopted  by  the  Office  of 
Personnel  Management  (OPM)  holds  that  tests  are  biased  when  those  with  equal 
chances  of  success  on  the  job  have  unequal  chances  of  being  selected  for  the 
job.  This  definition  relies  solely  on  valid  measures  of  aptitude,  achievement 
and  experience;  maximizes  productivity  and  provides  equal  opportunity  consistent 
with  merit  principles.  The  statistical  model,  upon  which  the  Uniform  Guidelines 
appear  to  be  based,  was  stated  by  Cleary  (33).  Basically,  this  model  requires 
the  regression  line  which  predicts  the  criterion  (e.g.  performance)  from  "scores" 
be  the  same  for  all  cultural  groups  and,  absent  this,  separate  regression  lines 
should  be  used  as  a  basis  for  selection  decisions.  Statistical  tests  developed 
by  Gulliksen  and  Wilkes  (34)  provide  a  procedure  which  meets  the  Cleary  model. 
Basically,  three  independent  and  sequential  tests  are  involved: 

(1)  population  variances  (standard  error  of  estimates)  are  equal 
(non-significant  differences),  then, 

(2)  slopes  of  the  population  regression  lines  are  equal  (non¬ 
significant  differences) ,  then 

(3)  intercepts  of  the  population  regression  lines  are  equal 
(non-significant  differences). 

If  each  of  the  three  statistical  tests  is  met,  the  regression  lines  are  the  same 
and  the  "fairness"  definition  of  "Qualified  Individualism"  is  satisfied  in  that 
"scores"  used  for  selection  predict  the  criterion  equally  for  the  various  groups 
considered.  On  the  other  hand,  if  a  statistically  significant  difference  is 
found  at  any  point  in  the  sequential  analysis,  the  test  is  considered  to  treat 
the  groups  unequally. 
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In  examining  the  new  ATC  test  battery  in  accordance  with  the  "Qualified 
Individualism"  model  of  fairness,  test  scores  and  criterion  data  obtained 
for  the  sample  of  953  ATC  trainees  attending  the  FAA  Academy  during 
June  1978  through  December  1978  were  used  in  the  analysis.  The  number  of 
Hispanics  (23  total — 7  with  OPM  test  data)  was  too  small  for  analysis. 
There  were  no  American  Indians  in  this  sample.  There  was  no  evidence  of 
adverse  impact  on  selection  of  Asians  with  the  new  ATC  test  battery.  Con¬ 
sequently,  the  analysis  for  test  fairness  was  completed  with  respect  to 
men  and  women  or  for  White  and  Black  groups. 

Sample  sizes  varied  depending  on  the  availability  of  data  on  the  two  OPM 
tests  (OPM  157  and  24)  included  in  the  new  ATC  test  battery.  Test  and 
criterion  data  were  available  for  the  following  ATC  trainees  (Table  117). 

TABLE  117 

DISTRIBUTIONS  BY  SELECTED  SAMPLES 


Men 

Woaen 

While 

Black 

(N-800) 

(N-839) 

(N-81) 

S maple 

Staple 

Staple 

Staple 

Test 

N 

N 

S 

N 

MCAT  1 

790 

137 

828 

79 

MCAT  TOTAL 

790 

137 

828 

79 

OPM  157 

509 

66 

539 

38 

OPM  24 

509 

66 

539 

38 

In  conducting  the  analysis  of  test  fairness,  the  tests  were  weighted  in 
accordance  with  the  values  derived  for  the  1978  ATC  trainees.  These 
weights  are: 

2  x  MCAT  1  (Total  Right  Scores)')  MCAT  Total  x  2 
2  x  MCAT  2  (Total  Right  Scores) > 

1  x  OPM  157  (Total  Right  Scores) 

1  x  OPM  24  (Total  Right  Scores) 

The  test  for  significance  for  all  analyses  of  fairness  was  the  .05  level  of 
confidence. 

Since  almost  all  of  the  1978  ATC  trainee  sample  had  test  scores  available 
on  the  two  parallel  forms  of  MCAT,  an  analysis  of  fairness  of  the  MCAT  by 
itself  was  completed.  Table  118  shows  the  results  for  the  first  form  of 
the  test  administered  (MCAT  1)  and  the  total  score  for  both  forms  (MCAT  1 
plus  MCAT  2)  by  race  and  sex.  The  test  means  and  standard  deviations  are 
based  on  raw  test  scores.  Since  MCAT  was  the  only  test  used  in  this  analysis, 
the  raw  scores  were  not  weighted.  The  criterion  value  is  based  on  a  mean  of 
"0"  with  a  standard  deviation  of  "1"  and  reflects,  in  effect,  the  percent 
of  trainees  who  passed  the  Laboratory  training;  the  validity  coefficient  (r) 
is  the  correlation  value  of  the  test  with  the  pass  or  fail/withdraw  criterion. 
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In  terms  of  sample  sizes  used  in  this  analysis,  the  137  women  comprise 
14.8  percent  of  the  combined  sample  of  men  and  women  (927)  compared  to 
15.7  percent  of  the  7,894  ATC  trainees  who  were  women  hired  from  January 
1976  through  October  1980.  The  79  Black  trainees  represent  8.7  percent 
of  the  combined  sample  of  the  White  and  Black  group  (907)  compared  to 
8.3  percent  of  the  7488  White  and  Black  ATC  trainees  hired  during  this 
same  period  (see  Table  123). 


For  men  and  women,  the  test  of  fairness  shows  no  significant  differences 
(NS).  For  the  White  and  Black  groups,  the  difference  in  the  intercept  of 
the  regression  line,  was  statistically  significant  (SIG)  for  the  first 
MCAT  test  administered  but  differences  in  the  standard  error  and  the  slope 
of  the  regression  line  were  not  significant  (NS) .  When  the  total  scores 
for  both  MCAT  forms  administered  were  analyzed,  there  were  no  significant 
differences  between  the  two  groups.  This  analysis  indicates  that  the  total 
MCAT  scores  predict  the  ATC  laboratory  pass  or  fail/withdraw  criterion 
equally  for  men  and  women  and  for  the  Black  and  White  groups. 

The  next  step  was  to  develop  different  test  battery  combinations  with  the 
MCAT  test  scores  and  0PM  157,  0PM  24,  and  OKI  test  scores  and  analyze 
these  various  test  batteries  for  significant  differences  between  men  and 
women  and  the  White  and  Black  groups.  The  intercorrelations  between  the 
various  test  combinations  used  and  their  correlation  with  the  pass  or  fail/ 
withdraw  criterion  are  provided  for  each  group  in  Table  93.  Since  a  number 
of  trainees  did  not  have  test  scores  on  0PM  157  (and  0PM  24) ,  the  sample 
sizes  were  reduced  as  shown  in  Table  JJ9.  The  correlations  shown  are  based 
on  weighted  test  scores  where  appropriate  using  raw  test  battery  scores  (RS) 
as  well  as  transmuted  scores  (TS) .  The  correlations  which  combine  a  test 
battery  with  OKT  use  both  the  raw  OKT  score  (RS)  as  well  as  OKT  extra  points 
(PTS)  derived  from  the  raw  OKT  score  (i.e.,  0,  3,  5,  10,  15  points). 

The  descriptive  statistics  for  the  samples  used  in  the  fairness  analysis  are 
also  provided  in  comparison  to  the  total  sample  for  each  group  as  was  given 
in  Table  84. 
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TABLE  119 

TEST  FAIFNESS  ANALYSES  OF  TEST  BATTERY  OPTIONS 

—MEN -  - WOMEN - 

(Table  84  (Table  84 


N 

MEAN 

SD 

N 

MEAN 

SD 

N 

MEAN 

SD 

N 

MEAN 

SD 

MCAT  1 

509 

37.6 

6.6 

800 

37.4 

6.9 

66 

33.4 

8. 1 

141 

34.1 

7.8 

MCAT  2 

509 

43.3 

5.  7 

800 

42.9 

5.8 

66 

40.  7 

6.  7 

141 

40.7 

6.4 

MCAT  TOT 

509 

80.9 

11.0 

800 

— 

— 

66 

74.1 

13.4 

141 

— 

- 

OPM  157 

509 

38.8 

6.2 

515 

38.8 

6.2 

66 

39.9 

6.0 

67 

39.8 

6.0 

0PM  2* 

509 

46.7 

6.5 

515 

46.6 

6.5 

66 

47.3 

7.1 

67 

47.2 

7.1 

OKT  (RS) 

509 

64.1 

15.8 

800 

64.9 

15.5 

66 

56.5 

14.4 

141 

57.7 

15.1 

- MLN  — 

- - - 

(N-509) 

(1)  (2) 

(3) 

(4) 

(5) 

(6) 

(?) 

(8) 

(9) 

(10) 

(11) 

(12) 

(13) 

(TS) 

(TS) 

MCAT 

MLAT 

MCAT 

MCAT 

MCAT 

TOT. 

MCAT 

TOT. 

MCAT 

MCAT 

TOT. 

TOT. 

TOT. 

157 

TOT. 

157 

TOT. 

TOT. 

157 

157 

157 

24 

157 

24 

Pass 

MCAT  OPM 

OPM 

OKT 

157 

1S7 

OKT 

OKT 

24 

OKT 

24 

OKT 

Fall 

TOT.  157 

24 

(»5) 

(RS) 

(TS) 

IRS) 

(PTS) 

(RS) 

JJElL 

iI£L 

UP 

(1) 

MCAT  TOT. 

.24 

.33 

.03 

.97 

.97 

.83 

.82 

.95 

.86 

.95 

.82 

.  38 

(2) 

OPM  157 

— 

.09  - 

.16 

.48 

.48 

.32 

.32 

.  4  b 

.32 

.45 

.  30 

.  16 

(3) 

OPM  24 

-  - 

-.23 

.  32 

.32 

.14 

.15 

.53 

.35 

.53 

.  33 

.12 

(4) 

OKT  (RS) 

— 

-.02 

-.02 

.54 

.44 

-.07 

.46 

-.07 

.41 

.  21 

(5) 

MCAT  TOT 

♦ 

15?(RS) 

— 

1 .00 

.83 

.83 

.  97 

.86 

.97 

.82 

.  38 

<b) 

MCAT  TOT 

♦ 

157(TS) 

— 

.83 

.83 

.97 

.86 

.97 

.82 

.  38 

(7) 

MCAT  TOT 

♦ 

157  ♦  OKT  (RS) 

— 

.94 

.  78 

.98 

.  78 

.92 

.4c 

(8) 

MCAT  TOT 

♦ 

157(TS)  ♦  OKT(PTS) 

— 

.78 

.92 

.78 

.98 

.44 

<9) 

MCAT  TOT 

♦ 

157  +  24(RS) 

— 

.85 

1.00 

.82 

,  37 

(10) 

MCAT  TOT 

♦ 

157  +  24  +  OKT (RS) 

— 

.85 

.  9* 

.4* 

Ul) 

MCAT  TOT 

■f 

157  ♦  24 (TS) 

— 

.82 

.  37 

(12) 

MCAT  TOT 

♦ 

157  ♦  24 (TS) 

♦  OKT(PTS) 

_ 

.44 

CD)  Criterion  (Lab  Paaa  or  F/VD) 
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TABIJ'  1 19  (Gontmue-di 


- -WOMi S- 

— 

— 

'* 

(1)  I-1)  U) 

(4) 

<■) 

<N-6» ) 
it) 

(7) 

(6) 

(V) 

I  JO) 

(11) 

<12  ) 
(TS) 

O  i) 

(lb) 

Ml  AT 

ML  AT 

Ml  AT 

MCA  r 

MCA  I 

TOT. 

MCAT 

Tor 

MCAT  OPM  OPM 

TOT.  IV)  _2h. 

OKT 

(HS) 

MCAT 

TOT. 

is; 

i5SI 

MtAl 

TOT. 

is; 

TOT. 

1  s  7 
OKT 

Tul  ■ 
JS7 
OKT 

TOT  . 

1S7 

24 

L*±l 

]  5 ' 

24 

OKT 

(KS) 

TOI. 

157 

24 

(TS) 

157 

24 

OKT 

imi. 

Pass 

Fall 

WO 

(1)  HOT  TOT. 

.2)  .0- 

.  17 

.  ^8 

.96 

.0* 

.  9; 

.  93 

.89 

.95 

.90 

.  33 

12)  OPM  157 

— - 

.Ob 

i  y 

.  3«. 

.  4  3 

.  -J 

.43 

.  35 

.  *«> 

( 3 )  OPM  24 

—  - 

-.25 

.  Ob 

.06 

-  .06 

.01 

.  29 

.  IS 

.  29 

.20 

.08 

(4)  OCT  (BS) 

... 

.  1  7 

.17 

.56 

.41 

.11 

.51 

.a 

.  37 

.  31 

(5)  MCAT  TOT  ♦ 

1S7(RS) 

... 

1.00 

.91 

.93 

.97 

.91 

.97 

.91 

.  39 

(6)  MOT  TOT  ♦ 

I57(TS) 

... 

.91 

.9) 

.97 

.91 

.97 

.  91 

.39 

(  7)  MOT  TOT  ♦ 

157  ♦  OCT(AS) 

— 

.95 

.86 

.  98 

.86 

.92 

.  46 

(8)  MOT  TOT  ♦ 

157(TS)  ♦  OCT(PTS) 

... 

.89 

.  93 

.69 

.98 

.41 

(9)  MOT  TOT  ♦ 

157  ♦  24(RS) 

— 

.91 

1.00 

.92 

.39 

(10)  MOT  TOT  ♦ 

157  ♦  24  ♦  OCT(RS) 

— 

.91 

.95 

.47 

(11)  MOT  TOT  ♦ 

157  ♦  24(TS) 

— 

.92 

.  39 

(12)  MOT  TOT  * 

157  ♦  24<TS)  ♦  OKT(PTb) 

— 

..2 

(13)  Criterion 

(Lab  Pas*  or  F/WD) 

— 

N 

MEAN 

- WHITE¬ 
ST) 

N 

(Table 

MEAN 

84 

SD 

N 

MEAN 

SD 

•BLACK — 

l 

N 

(Table 

MEAN 

86 

SD 

MCAT  X 

539 

37.7 

6.6 

839 

37.6 

6.7 

38 

29.7 

7.8 

81 

29.8 

7.9 

MCAT  2 

539 

43.6 

5.5 

839 

43.2 

5.5 

38 

36.1 

7.5 

81 

36.6 

6.0 

MCAT  TOT 

539 

81.2 

10.6 

839 

— 

— 

38 

65.8 

14.0 

81 

-- 

— 

OPM  157 

539 

39.1 

6.2 

5*5 

39.1 

6.2 

38 

35.4 

6.1 

39 

35.3 

6.1 

OPM  24 

539 

46.6 

6.6 

545 

46.9 

6.5 

38 

44.  7 

6.6 

39 

44.2 

7.1 

OCT  (RS) 

539 

63.1 

15.9 

839 

63.9 

15.8 

38 

65.5 

13.6 

81 

66.2 

1  3.  1 
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TABLE  119  ( ODnt  3  nuc-d 


WHITE- 

IN-539) 


(1) 

U) 

l  >> 

(4) 

(5) 

(6) 

MCAT 

TUT 

uPM 

IS 7 

Oi'M 
'  4 

OKT 

mcaT 

TOT. 

157 

(RS) 

Hi.  A1 
TUT. 
157 

Alii 

(1) 

HCAT  TOT. 

--- 

. :  3 

.  -  f* 

.07 

.  97 

.97 

(2) 

OPH  157 

— - 

;  .t 

u 

.48 

.48 

(3) 

OPM  24 

— 

- .  24 

.  28 

.28 

(4) 

OKT  (RS/ 

.03 

.03 

(5) 

HCAT  TOT 

♦  15?<KS) 

--- 

1.00 

(6 )  MCAT  TOT  ♦  157<TS) 

(7)  HOT  TOT  4  157  ♦  OlCT(RS) 

(8)  HCAT  TOT  4  157 (TS)  4  OKT(PTS) 

(9)  HCAT  TOT  4  157  4  24(RS) 

(10)  HCAT  TOT  ♦  15)  +  20  OKT(RS) 

(11)  HCAT  TOT  ♦  157  ♦  24(TS) 

(12)  HCAT  TOT  ♦  157  4  24(TS)  ♦  OKT(PTS) 

(13)  Criterion  (Lab  Pass  or  F/WO) 


(7) 

c«> 

(9) 

ciu) 

(ID 

( 1 . ) 
ITS) 

Kl  3) 

(TS) 

hi  AT 

HLAl 

M(  A T 

HCAT 

MCA! 

TuT. 

HCAT 

Tor 

tut  . 

TUT. 

TUT  . 

157 

TOT. 

15) 

1  s  ? 

1  S  7 

157 

24 

15  7 

24 

fat.* 

i  >  / 

OKT 

ASiL 

OKI 

(PTs) 

24 

IKS). 

OKT 

(RS) 

24 

Alii 

OKT 

IPTbl 

fa  l  1 
Vfl) 

.83 

.  9* 

.85 

.  9* 

.81 

.  Si 

.31 

.3: 

.45 

.  32 

.45 

.  31 

.  i  7 

.09 

.  10 

.50 

.31 

.50 

.30 

.o; 

.58 

.  4fc 

-.03 

.50 

-.Oj 

.-4 

.23 

.83 

.  6* 

.97 

.  8b 

.  97 

.82 

.  3+4 

.83 

.82 

.97 

■  8b 

.97 

.82 

.34 

_ 

.94 

.77 

.98 

.  77 

.92 

.41 

— 

.77 

.92 

.  77 

.98 

.40 

— 

.85 

1  .00 

.81 

.  3i 

... 

.85 

.94 

.  <4  1 

— 

.81 

.33 

_ 

.40 

BLACK* 

(N-)8) 


(1)  (2) 

(3) 

(4) 

(5) 

(b) 

(7) 

(8) 

(9) 

(10) 

(U) 

cl.) 

(TS; 

(13) 

ITS) 

MCAT 

Him 

MCAT 

Mc\r 

MCA1 

TOT. 

MCAT 

to: 

HCAT  OPM 

TOT.  157 

OPM 

24 

OKT 

(RS) 

HCAT 
TOT  . 
157 
(RS) 

HCAT 

TOT. 

15/ 

Alii 

TOT. 

157 

OKT 

(RS) 

TOT. 

1S7 

OKT 

ALtii 

TOT. 

157 

24 

(RS) 

15? 

24 

OKT 

(RS) 

TOT. 

157 

24 

(TS) 

157 

24 

OKT 

imi 

Pubs 
Fj  11 

UP 

(1)  HCAT  TOT.  -  *  28 

.17 

.27 

.98 

.98 

.90 

.92 

.96 

.91 

.96 

.91 

.60 

(2)  OPH  157 

-.11 

-.11 

-.06 

-.06 

-.10 

-.18 

-.09 

-.12 

-.09 

-.19 

.07 

(3)  OPH  24 

— 

-.09 

.15 

.15 

.08 

.09 

.  37 

.27 

.  37 

.  23 

.35 

(4)  OKT  (RS) 

— 

.25 

.25 

.62 

.49 

.22 

.58 

.22 

.47 

.24 

(5)  HCAT  TOT  ♦  157(RS) 

— 

1.00 

.92 

.92 

.97 

.91 

.  .97 

.  90 

.64 

(6)  HCAT  TOT  ♦  157 (TS) 

— 

.92 

.  92 

.97 

.91 

.  97 

.90 

.64 

(7)  HCAT  TOT  ♦  157  ♦  OKT(RS) 

— 

.95 

.88 

.98 

.88 

.93 

.62 

(8)  HCAT  TOT  4-  1 S 7 (TS)  4  OKT(PTS) 

— 

.89 

.94 

.89 

.99 

.61 

(*)  HCAT  TOT  4  157  4  24<RS) 

— 

.92 

1.00 

.91 

.68 

(10)  HCAT  TOT  ♦  15  7  ♦  24  ♦  OKT(RS) 

— 

.92 

.95 

.67 

(11)  HCAT  TOT  4  157  4  24(TS) 

— 

.91 

.68 

(12)  HCAT  TOT  4  157  4  24<TS)  4 

o 

* 

-4 

3 

) 

.64 

(13)  Criterion  (Leb  Pern  or  F/WD)  J44 


The  first  test  battery  analyzed  for  fairness  consisted  of  MCAT  Total  and 
OPM  157  raw  scores.  In  this  analysis,  MCAT  Total  was  weighted  by  "2"  and 
0PM  157  weighted  by  "1"  in  accordance  with  the  weights  previously  derived. 

Then  the  OKT  raw  scores  (weighted  "1")  were  combined  with  the  MCAT  and 
OPM  157  raw  scores  and  this  combination  separately  evaluated.  The  results 
for  each  sex  and  race  group  are  provided  in  Table  120.  It  should  be  noted 
that  at  this  point,  a  "passing"  score  for  the  combined  MCAT  and  OPM  157 
test  scores  was  not  established.  Consequently,  the  addition  of  the  OKT  raw 
score  was  independent  of  whether  or  not  the  ATC  trainee  "passed"  the  MCAT/OPM 
157  test  battery  and  also  independent  of  extra  credit  points  (0,  3,  5,  10,  15) 
which  would  be  derived  from  the  raw  OKT  scores. 


The  combination  of  MCAT  and  OPM  157  raw  test  scores  show  that  this  test 
battery  predicts  the  ATC  laboratory  pass  or  fail/withdraw  criteria  equally 
for  men  and  women  and  for  Black  and  White  groups.  The  same  results  were 
obtained  when  MCAT,  OPM  157  and  OKT  raw  test  scores  were  analyzed  separately. 

Use  of  raw  scores  in  analyzing  the  test  battery  comprised  of  MCAT,  OPM  157 
and  OKT  does  not  take  into  account  the  proposed  use  of  OKT  in  the  selection 
process.  Since  OKT  is  a  job  specific  test,  it  is  not  intended  for  use  in 
the  initial  "pass/fail"  determination  of  appointment  eligibility  for  ATC 
applicants.  Its  use  would  be  limited  to  determining  if  those  applicants 
who  passed  the  test  battery  (in  this  case  MCAT  and  OPM  157)  would  also  be 
given  extra  credit  for  air  traffic  control  related  knowledge  as  measured  by 
OKT.  Further,  the  amount  of  extra  credit  (0,  3,  5,  10,  15  points)  would  be 
based  on  the  raw  OKT  score  with  no  credit  for  raw  scores  below  65  and  maximum 
credit  (15  points)  for  raw  OKT  scores  of  60  or  above. 

In  order  to  evaluate  the  fairness  of  a  test  battery  in  combination  with  OKT 
in  relation  to  its  intended  use,  it  is  necessary  to  (1)  establish  a  "passing" 
score  on  the  test  battery  for  an  ATC  applicant  group,  (2)  transmute  the 
raw  test  battery  score  in  relation  to  the  passing  score,  and  (3)  add  the 
appropriate  extra  credit  points  based  on  OKI  raw  score  for  those  who  passed 
the  test  battery. 
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In  order  to  transmute  the  total  raw  score  for  the  ATC  trainee  sample,  it  was 
necessary  to  estimate  the  mean  score  which  would  have  been  obtained  by  an 
ATC  applicant  .group  on  the  ATC  test  battery  and  equate  this  mean  score  to 
a  passing  score  of  70.  Fortunately,  the  tests  used  in  the  analysis  of  the 
ATC  trainee  sample  were  identical  to  the  tests  taken  by  the5,931ATC  appli¬ 
cants  in  1978  except  for  the  second  form  of  MCAT  which  was  not  administered 
to  the  ATC  applicants.  Consequently,  given  the  mean  scores  on  each  test 
which  were  available  for  both  the  ATC  trainees  and  the  ATC  applicants,  it  is 
possible  to  estimate  a  mean  for  the  ATC  applicants  on  the  second  MCAT  form. 
By  combining  this  estimated  mean  for  "MCAT  2"  with  the  actual  mean  scores 
obtained  by  the  ATC  applicants  on  the  other  tests  and  weighting  them,  an 
estimated  mean  score  of  161.2  for  the  1978  ATC  applicant  group  on  a  test 
battery  comprised  of  MCAT  Total  and  0PM  157  was  derived  as  shown  below: 


1978 

ATC  Applicants 


1978 

ATC  Trainees 


MEAN  x 

WT.  - 

TOTAL 

MEAN 

x  WT. 

“  TOTAL 

MCAT  1 

30.6 

2 

61.2 

36.9 

2 

73.8 

* (MCAT  2 

35.6 

2 

71.2) 

42.6 

2 

85.2 

0PM  157 

28.8 

1 

28.8 

38.8 

1 

38.8 

Est. 

Mean  Raw 

Score 

161.2 

197.8 

(1) 

ATC  Applicant  Mean 

-  MCAT  1 

30.6 

ATC  Trainee  Mean  - 

MCAT  1 

36.9 

(2) 

ATC  Trainee  Mean  - 

MCAT  2 

42.6 

ATC  Trainee  Mean  - 

MCAT  1 

36.9 

Difference 

5.7 

*(3)  Est.  MCAT  2  Mean  for  Applicant  Group  *»  35.6 

(6.0  x  .829  -  5.0  +  30.6  «  35.6) 


Since  the  correlation  between  the  two  differently  weighted  ATC  tests  for 
the  ATC  trainee  sample  was  .93  (Table  98),  an  estimated  mean  score  for  the 
ATC  applicant  group  of  159.5  was  also  derived  using  the  ratio  of  the  mean 
scores  for  the  1978  applicant  and  ATC  trainee  groups  for  the  common  weighted 
tests  as  shown  on  the  next  page. 


147 


MEAN 


(1) 

Actual  Applicant  Group  Mean  (N  -  6000) 

(4  x  MCAT  1  +  2  x  157  ) 

181.6 

(Table 

43) 

(2) 

Actual  Mean  ATC  Trainee  Group  (N  ■  592) 

(4  x  MCAT  1  4-  2  x  157) 

225.2 

(Table 

69) 

(3) 

Actual  Mean  ATC  Trainee  Group  (N  -  592) 

(2  x  MCAT  1  +  2  x  MCAT  2  +  1  x  157) 

197.8 

(Table 

69) 

(4) 

197  ft  X 

- —  x  —  ■  159.5  Est.  MEAN  for  Applicant  Group  on  ATC 

225.2  181.6  (2  x  MCAT  !  +  2  x  MCAT  2  +  1  x  157) 

Test  Weighted 

Given  these  two  estimated  mean  scores  for  an  applicant  group  on  the  ATC  test 
battery  comprised  of  MCAT  and  OPM  157,  a  mean  value  of  160  was  used  as  a 
"passing"  score  and  equated  to  a  transmuted  score  of  70.  The  total  raw 
weighted  scores  for  each  ATC  trainee  on  MCAT  and  OPM  157  were  then  transmuted 
by: 


Ts 


(RS  -  160)  x  30 
MAX  RS  -  160 


)  +  70 


RS  *  Total  weighted  raw  score 
MAX  RS  -  255.0 
MEAN  RS  -  160.0 

An  analysis  of  fairness  was  completed  using  the  obtained  transmuted  test 
battery  scores  for  the  ATC  trainees.  A  separate  analysis  was  completed  using 
the  transmuted  scores  plus  extra  credit  points  (0,  3,  5,  10,  15)  derived 
from  OKT  raw  scores  for  those  ATC  trainees  who  "pass"  the  test  battery  with 
scores  of  70  or  more.  The  results  are  provided  in  Table  121. 


148 


TAPLF  121 

ANALYSIS  OF  TEST  FAIRNESS 


The  combination  of  transmuted  test  battery  scores  for  MCAT  and  OPM  157  plus 
extra  credit  points  for  air  traffic  control  related  knowledge  based  on  OKI 
scores  replicates  the  operational  use  of  the  selection  tests.  The  analysis 
of  fairness  provided  in  Table  1-1  shows  that  this  test  battery  including 
OKT,  predicts  the  AIC  laboratory  pass  or  fail/withdraw  criterion  equally  for 
men  and  women  and  for  Black  and  White  groups. 

The  final  step  in  the  fairness  analysis  was  to  examine  the  entire  test  battery 
comprised  of  MCAT,  OPM  157  and  OPM  24  with  and  without  extra  credit  for  air 
traffic  control  related  knowledge  based  on  OKT  scores.  The  results  are  pro¬ 
vided  in  Table  123.  In  order  to  add  OKT  extra  credit  points,  it  was  again 
necessary  to  transmute  the  total  raw  score  for  MCAT,  OPM  157  and  OPM  24  and 
to  establish  a  passing  raw  score  which  was  equated  to  a  transmuted  score  of 
70  as  shown  below: 


1978 

1978 

ATC  Applicants 

— 

AIC  .Trainees. 

TEST 

MEAN  x 

WT  - 

TOTAL 

MEAN 

x  WT 

=  TOTAL 

MCAT  1 

30.6 

2 

61.2 

36.9 

2 

73.8 

(MCAT  2 

35.6 

2 

71.2) 

42.6 

2 

85.2 

OPM  157 

28.8 

1 

28.8 

38.8 

l 

38.8 

OPM  24 

40.6 

1 

40.6 

46.6 

l 

46.6 

Est. 

Mean  Raw  Score 

201.8 

244.4 

*See  page  143  for  derivation  of  estimated  mean  score  for  MCAT  2. 

Again,  an  estimated  mean  raw  score  for  the  applicant  group  was  derived  using 
a  ratio  method  as  follows: 


(1)  Actual  Applicant  Mean  (N  »  6000) 

(4  x  MCAT  1  +  2  x  157  +  1  x  24) 

(2)  Actual  Mean  ATC  Trainee  Group  (N  *  592) 

(4  x  MCAT  1  +  2  x  157  +  1  x  24) 

(3)  Actual  Mean  ATC  Trainee  Group  (N  «  592) 

(2  x  MCAT  1  +  2  x  MCAT  2  +  1  x  157  +  1  x  24) 


222.3  (Table  62) 

272.6  (Table  83) 

245.6  (Table  83) 


(4)  245.6 

272.6 


222.3 


200.3  Est.  Mean  for  Applicant  Group  on  ATC  Test 
Weighted 

(2  x  MCAT  1  +  2  x  MCAT  2  +  1  x  157  +  1  x  24) 


Given  these  two  estimated  mean  scores  for  an  applicant  group  (201.8  and  200.3) 
on  the  ATC  test  battery  comprised  of  MCAT,  OPM  157  and  OPM  24,  a  mean  value 
of  200  was  used  as  a  "passing"  score  and  equated  to  a  transmuted  score  of  70. 
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The  total  raw  weighted  scores  for  each  ATC  trainee  on  MCAT,  OPM  157  and 
0PM  24  were  then  transmuted  by: 


TS 


(RS  -  200)  x  30  .  n 
MAX  RS  -200  ' 


Rg  •  Total  weighted  raw  score 

MAX  RS  -  306 
MEAN  RS  -  200 

Table  122  shows  the  results  of  the  fairness  analysis  on  this  ATC  test  battery 
(for  both  raw  and  transmuted  scores)  and  for  the  ATC  test  battery  transmuted 
scores  in  combination  with  extra  credit  points  based  on  0KT  for  those  trainees 
who  scored  70  or  more  on  the  test-  battery . 


As  shown  in  Table  122,  there  were  no  significant  differences  on  the  ATC  test 
battery  between  men  and  women.  However,  the  addition  of  OPM  24  test  scores 
(Arithmetic  Reasoning)  to  the  scores  of  MCAT  and  OPM  157  resulted  in  the  esti¬ 
mated  standard  error  (population  variances)  between  the  Black  and  White  groups 
being  statistically  significant.  The  statistical  tests  developed  by  Gulliksen 
and  Wilkes  (34)  require  that  if  a  significant  difference  is  found  at  any  of 
the  three  steps  in  the  analysis,  no  further  tests  are  made  since  the  regres¬ 
sion  lines  used  for  predictions  are,  by  definition,  unequal. 
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The  face  that  the  regression  lines  for  the  White  and  Black  groups  are  statis¬ 
tically  unequal,  by  itself,  does  not  identify  the  practical  implications  for 
a  selection  procedure.  Under  the  Cleary  model  of  test  fairness,  if  tests  are 
"biased"  (unequal) ,  an  alternative  solution  is  to  use  the  separate  regression 
lines  for  selection  decisions.  This  solution,  however,  explicitly  introduces 
race,  sex,  or  ethnic  group  (in  this  case  race)  as  a  predictor  in  the  selection 
process.  This  is  incompatible  with  the  ethical  position  of  "Qualified 
Individualism"  and  lav 

In  these  circumstances,  a  single  regression  line  derived  by  combining  the 
groups  typically  has  been  used  as  the  basis  for  selection  procedures.  The 
effect  of  this  is  to  "overpredict"  the  criterion  (e.g.,  performance)  for 
that  group  which  scored  lower  on  the  test  and  had  a  lower  mean  on  the  cri¬ 
terion  measure  <  >-)•  In  the  analysis  of  this  ATC  test  battery,  use  of  a 
single  regression  line  obtained  by  combining  the  White  and  Black  groups  (or 
the  total  sample)  would  "bias"  the  selection  procedure  in  favor  of  the  Black 
group.  This  result  conforms  with  the  general  findings  in  the  published 
literature  referred  to  by  the  General  Accounting  Office  in  their  review  of 
Federal  selection  tests  OO. 

Table  122  also  shows  that  when  extra  credit  points  based  on  OKI  are  added  to 
the  transmuted  scores  for  this  ATC  test  battery,  there  were  no  significant 
differences  between  men  and  women  or  the  Black  and  White  groups.  While 
this  indicates  that  the  combination  of  the  test  battery  and  OKT  points  pre¬ 
dicts  the  ATC  laboratory  pass  or  fail/withdraw  criterion  equally  for  all 
groups,  the  test  battery  itself  would  be  used  to  establish  initial  appoint¬ 
ment  eligibility  for  applicants.  The  fact  that  addition  of  OKT  points  results 
in  equal  treatment  for  those  who  pass  the  test  does  not  adequately  address 
the  unequal  treatment  resulting  from  the  use  of  OPM  24  under  the  Uniform 
Guidelines.  Given  that  OPM  24  results  in  unequal  prediction  of  the  pass  or 
fail/withdraw  criterion  between  the  White  and  Black  groups,  it  should  not 
be  used  as  part  of  the  ATC  selection  battery.  From  a  practical  viewpoint, 
the  exclusion  of  OPM  24  has  essentially  no  impact  on  the  multiple  correla¬ 
tion  or  the  predictive  value  for  the  total  group  or  for  men  and  women  since 
its  contribution  to  the  prediction  of  the  performance  criterion  is  very 
small  (see  Tables  t)r>  ;>ml  96) . 

Alternative  Selection  Procedures.  During  the  course  of  these  research  studies, 
a  number  of  alternative  instruments  of  a  cognitive  and  non-cognitive  structure 
were  examined.  The  1972,  1977,  an^  1978  studies  of  full-performance,  develop¬ 
mental,  and  trainee  air  traffic  controllers  examined  the  relationship  between 
non-cognitive  instruments  (including  biographical  data  and  personality  tests) 
and  performance  criterion.  Use  of  biographical  data  as  an  alternative 
selection  procedure  was  examined  in  some  depth. 

In  May  1979,  the  FAA  proposed  to  the  Office  of  Personnel  Management  (OPM)  a 
demonstration  project  under  the  Civil  Service  Reform  Act  that  addressed 
recruitment  and  selection  of  women  and  minorities  in  the  air  traffic  control 
occupation  05).  This  proposal  involved  a  5-year  period  during  which  approxi¬ 
mately  half  of  the  new  ATC  hires  (750)  in  the  FAA  Southern  region  would  be 


selected  in  nonregister  order  as  a  control  group  and  the  other  half  (750) 
selected  in  the  regular  manner  from  the  0PM  register  to  form  a  noncontrol 
group.  Candidates  in  the  Southern  region  would  be  given  a  detailed  multiple 
choice  biographical  questionnaire  and  "profiled"  on  relevant  life  experience 
dimensions  and  grouped  into  life  profile  clusters.  The  life  profiles  would 
then  be  used  as  an  alternate  means  for  ranking  and  selection  of  candidates 
for  the  control  group.  Success  in  ATC  training  would  then  be  evaluated  for 
both  the  control  and  noncontrol  groups  as  a  means  validating  the  alternative 
selection  procedure  1  1  • 

After  review  of  the  proposal  and  the  biographical  questionnaire,  0PM  pointed 
out  a  number  of  concerns  on  "Job  Relatedness",  Privacy,  "Subjectivity",  and 
"Public  Relations"  for  most  of  the  questions  included  in  the  biographical 
form  (36).  On  December  4,  1979,  the  Deputy  Director,  0PM,  suggested  the  use 
of  numerical  scores  based  on  "life  profiles"  derived  from  empirical  data 
established  from  validity  studies.  It  was  also  pointed  out  that  the  "job 
relatedness"  issue  presented  a  difficult  problem  because  of  practical  and 
legal  requirements  <  '7). 

The  biographical  questionnaire  proposed  for  this  project  has  been  administered 
to  545  of  the  ATC  trainees  attending  the  FAA  Academy  during  the  period  1976- 
1977.  Information  on  sex,  race,  0PM  selection  scores,  ATC  training  laboratory 
composite  scores  and  training  pass/fail  status  was  also  available  for  this 
sample  of  545  students.  These  data  were  provided  to  the  Institute  for 
Behavioral  Research,  University  of  Georgia,  for  analysis.  Eight  factors  were 
identified  through  factor  analysis: 

o  Academic  factor 
o  Social  factor 
o  Child  Relationship  factor 
o  Initiative  factor 
o  IFR/VFR  Experience  factor 
o  Parental  Permissiveness  factor 
o  Physical/Sports  factor 
o  Socio/Economic  factor 

Based  on  these,  the  545  trainees  were  clustered  into  six  subgroups  with  similar 
life  experiences.  Because  of  the  weak  relationship  between  success  or  failure 
in  the  laboratory  training  and  the  subgroup  memberships,  the  eight  life  ex¬ 
perience  factors  were  used  to  predict  the  laboratory  score  using  regression 
and  stepwise  regression  analysis.  The  results  showed  that  the  factors  were 
not  good  predictors  of  the  laboratory  composite  score  which  determined  the 
pass/fail  status  of  ATC  trainees.  Generally  the  results  showed  that  ATC 
experience  and  tests  involving  mathematics  and  physical  sciences  would  yield 
better  predictions  of  success  in  AT  training.  The  report  (  1<I '  suggested 
that  using  a  sample  of  working  controllers  to  establish  similar  life  experi¬ 
ence  and  group  membership  might  prove  more  meaningful.  While  this  had  been 
planned  as  part  of  the  project  methodology  (36),  the  results  of  the  University 
of  Georgia  study,  the  0PM  concerns  regarding  validity  of  biographical  data  as 
a  selection  basis  and  cost  consideration  resulted  in  terminating  this  approach 
toward  alternative  selection  procedures. 
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As  pointed  out  in  conjunction  with  the  analysis  of  the  sample  of  953  ATC 
trainees  attending  the  FAA  Academy  in  1978  (page  101),  the  FAA  recognized 
the  difficulty  of  selecting  women  and  minorities  from  competitive  0PM 
registers  in  1968.  As  a  consequence,  the  Predevelopmental  ATC  program  was 
established.  In  1974,  Executive  Order  11813  provided  for  noncompetitive 
conversion  of  Cooperative  Education  students  and  this  authority  was  incor¬ 
porated  in  the  recruitment  and  selection  program  for  women  and  minorities 
in  air  traffic  and  other  FAA  occupations. 

These  alternative  selection  programs  have  been  and  will  continue  to  be  a 
major  vehicle  to  address  the  adverse  impact  on  women  and  minorities  as 
well  as  the  Federal  Equal  Opportunity  Recruitment  Program  (FEORP)  require¬ 
ments  established  by  the  1978  Civil  Service  Reform  Act. 

Table  123  shows  the  total  number  of  ATC  trainees  hired  from  January  1976 
through  October  1980;  those  hired  from  0PM  register  (competitive)  and  those 
hired  through  the  Predevelopmental  and  Co-op  program  (non-competitive)  by 
race  and  sex. 


TABLE  123 

ATC  TRAINEES  HIRES 
January  1976-Octoker  1980 


Croup 

Total 

Hirea 

*  of 

Total 

Ctwpecit  ive 
Hires 

X  of 

Croup 

Non-Competitive 

Hires 

X  of 

Men 

6653 

(  84.31) 

6218 

(  93. 5X) 

435 

(  6.5Z) 

Uoacn 

1241 

(  15.71) 

837 

i  67^4X)_ 

404 

(  32. 6X) 

Total 

7894 

(100.01) 

7055 

(  89. 4X) 

839 

(  10. 6X) 

Aaarican  Indian 

21 

(  .31) 

18 

(  85. 7X) 

3 

(  14. 3X) 

Aaian 

66 

(  .91) 

61 

(  92. 4X) 

5 

(  7.6X) 

Hispanic 

220 

(  2.8Z) 

134 

(  60. 9X) 

86 

(  39. IX) 

Black 

618 

(  7.9X) 

300 

(  48. 5X) 

318 

(  51. 5X) 

White 

6870 

(  88. IX) 

6442 

(  93. 8X) 

428 

(  6.2X) 

Total 

7795 

(100. 0Z) 

6955 

(  89. 2X) 

840 

(  10.81) 
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CONCLUSIONS  AND  RECOMMENDATIONS 


CONCLUSIONS:  The  various  research  studies  previously  discussed  support  the 
following  conclusions. 

•  The  experimental  ATC  test  battery  comprised  of  MCAT,  OPM  157, 
and  OPM  24  is  a  valid  and  statistically  significant  tool  for 
use  in  screening  applicants  for  the  ATC  occupation.  The  need 
for  screening  applicants  is  particularly  important  for  this 
occupation  because  of:  (1)  the  high  cost  of  training  to  the 
taxpayer;  (2)  the  large  number  of  applicants  in  relation  to 
the  relatively  few  vacancies  which  are  filled  each  year;  and 
(3)  the  fact  that  there  is  essentially  no  "self  screening"  on 
the  part  of  applicants  since  there  are  no  educational  or  spe¬ 
cialized  experience  requirements  for  appointment  eligibility 
at  the  entry  grade  levels  (GS-5  or  GS-7). 

•  The  use  of  the  Occupational  Knowledge  Test  (OKT)  in  conjunction 
with  the  experimental  ATC  test  battery  increases  the  predictive 
validity  of  the  selection  procedure.  It  provides  a  better 
method  of  establishing  the  applicant’s  knowledge  relevant  to 
the  ATC  occupation  as  compared  to  the  Rating  Guide  presently 
used  by  the  Office  of  Personnel  Management.  Further,  the 

use  of  OKT  allows  applicants  who  have  acquired  this  knowledge 
outside  of  the  specific  work  experiences  given  credit  under 
the  present  Rating  Guide  to  earn  extra  credit  for  competitive 
selection  consideration. 

•  The  use  of  the  experimental  ATC  test  battery  and  OKT  scores  for 
ranking  competitive  ATC  applicants  for  appointment  consideration 
and  selecting  those  with  the  highest  scores  first  is  a  valid  use 
of  the  applicant^  test  scores  since  it  significantly  increases 
the  probability  of  success  in  the  ATC  occupation  as  measured  by 
passing  the  Initial  ATC  Qualification  Training  Program.  The 
addition  of  veterans  preference  points  to  an  applicant's  score 
on  the  ATC  test  battery  is  not  a  valid  predictor  of  success  in 
the  ATC  occupation. 

•  The  statistical  analyses  encompassed  a  large  number  of  different 
experimental  tests  including  the  five  OPM  tests  presently  used 
for  scoring  ATC  applicants.  The  resulting  experimental  ATC 
test  battery  (MCAT,  OPM  157,  and  OPM  24)  was  derived  from 
multiple  regression  analysis.  The  tests  were  examined  with 
respect  to  both  unweighted  and  weighted  test  score  values. 

Only  the  OPM  157  test  of  the  present  five  OPM  tests  contributed 
to  the  multiple  correlation  (R)  predicting  the  ATC  training 
(performance)  criteria.  Weighting  the  tests  increased  the 
validity  of  the  ATC  test  battery. 
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•  Analysis  of  the  utility  of  the  ATC  test  battery  indicates  a 
significant  potential  for  reducing  overall  cost  of  ATC 
training.  The  new  test  is  effective  in  identifying  those 
applicants  who  pass  the  present  OPM  ATC  test  but  have  high 
fail/withdraw  rates  in  the  Initial  ATC  Qualification  training 
program.  Operational  use  of  the  new  ATC  test  battery  would 
require  more  frequent  administration  (perhaps  three  times  a  year) 
in  order  to  obtain  more  applicants  with  Earned  Rating  scores 
above  85  as  compared  to  the  present  OPM  test. 

•  Establishing  the  passing  score  on  the  new  ATC  test  battery  at 
the  approximate  mean  score  for  an  ATC  applicant  group  is 
supported  by  the  analysis  of  fail/withdraw  rates  by  score  range 
groups.  Setting  a  higher  passing  score  (for  example  a  transmuted 
score  of  80)  would  significantly  increase  the  adverse  impact  on 
women  and  minorities  as  well  as  make  it  extremely  difficult  to 
recruit  women  and  minorities  through  either  competitive  or  non¬ 
competitive  selection  procedures. 

•  The  present  OPM  ATC  test  battery  and  the  experimental  ATC  test 
battery  both  have  an  adverse  impact  on  some  minority  groups, 
particularly  Blacks.  When  extra  credit  for  veterans  preference  and 
ATC  related  knowledge  or  experience  is  granted  to  those  applicants 
who  pass  the  test  battery,  there  is  also  an  adverse  impact  on 
womer.  The  experimental  ATC  test  battery  has  a  somewhat  greater 
adverse  impact  on  selection  of  Women  and  Blacks  than  does  the  present 
OPM  ATC  test  battery.  The  veterans  preference  credit  is  a  signi¬ 
ficant  factor  in  the  adverse  impact  on  Women.  While  the  statis¬ 
tical  evidence  does  not  support  the  validity  of  veterans  preference 
in  relation  to  performance  criteria,  it  is  required  by  law. 

•  Given  the  evidence  of  adverse  impact  on  women  and  some  minority 
groups,  especially  Blacks,  analyses  of  test  fairness  were  com¬ 
pleted  on  the  various  components  of  the  experimental  test  battery 
and  the  OKI.  These  analyses  show  that  the  requirements  of  the 
Uniform  Guidelines  on  Employee  Selection  are  met  for  the  ATC  test 
battery  comprised  of  MCAT  and  OPM  157  and  for  MCAT ,  OPM  157  and 
OKE  extra  credit  points.  This  was  true  for  both  men  and  women 
and  the  White  and  Black  groups. 

•  The  Multiplex  Controller  Aptitude  test  (MCAT)  is  the  major  com¬ 
ponent  in  the  validities  obtained  with  the  new  ATC  test.  The 
analysis  of  fairness  for  MCAT,  by  itself,  included  a  sample  of 
men  and  women  as  well  as  White  and  Black  trainees  which  w.is  pro¬ 
portional  to  the  total  population  of  ATC  trainees  attending  the 
FAA  Academy  during  the  period  of  January  1976  through  October  1980. 
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•  The  addition  of  OPM  24  test  scores  to  the  MCAT  and  0PM  157  test 
battery  revealed  a  significant  difference  in  the  population 
variance  between  the  White  and  Black  groups  indicating  that  an 
ATC  test  battery  comprised  of  MCAT,  OPM  157  and  OPM  24  did  not 
meet  the  fairness  requirements  of  the  Uniform  Guidelines  for  the 
White  and  Black  groups.  The  effect  of  these  differences  would  be 
to  bias  the  test  battery  somewhat  in  favor  of  the  Black  group  if 
a  common  regression  line  for  the  two  groups  were  used.  However, 
there  'Were  no  significant  differences  between  men  and  women  on 
this  ATC  test  battery. 

RECOMMENDATIONS ;  Based  on  the  conclusion  derived  from  the  research  studies, 
the  following  recommendations  for  the  use  of  selection  tests  and  procedures 
to  qualify,  rate  and  rank  applicants  for  the  Air  Traffic  Control  occupation 
are  supported. 

•  The  ATC  Test  Battery  used  for  qualifying  applicants  for  placement 
on  competitive  OPM  registers  or  for  qualifying  applicants  for 
non-competitive  appointment  consideration  should  consist  of: 

(1)  The  two  forms  of  the  Multiplex  Controller  Aptitude  Test 
(MCAT)  with  the  total  correct  scores  weighted  by  a 
value  of  2. 

(2)  The  present  OPM  Test,  Abstract  Reasoning  and  Letter 
Sequence  scored  (R-1/4W)  weighted  by  a  value  of  1. 

(3)  Each  form  of  the  MCAT  has  55  questions  segmented  into 
two  parts;  20  minutes  is  allowed  for  Part  A  (27  or  28 
questions);  and,  15  minutes  allowed  for  Part  B  (27  or 
28  questions) .  Eight  minutes  is  also  allowed  for 
reading  test  directions  and  practice  problems.  There¬ 
fore,  a  total  of  43  minutes  is  required  for  administra¬ 
tion  of  each  MCAT  form  and  the  total  test  time  required 
for  both  MCAT  forms  is  1  hour  and  26  minutes.  The 
Abstract  Reasoning  and  Letter  Sequence  test  is  a  two- 
part  test;  each  part  has  25  questions.  Test  time  for 
Fart  A  is  15  minutes;  Part  B  is  20  minutes.  There  are 

9  sample  items  with  5  minutes  allowed  for  these  practice 
questions. 

(4)  Total  testing  time  for  the  recommended  ATC  Test  Battery, 
including  practice  and  test  familiarization  time,  is 

2  hours  and  6  minutes. 

•  The  composite  weighted  raw  score  on  the  test  battery  should  be  trans¬ 
muted  by  a  linear  transformation  of  the  distribution  so  that  the 
mean  raw  score  of  the  applicant  group  is  equated  to  a  passing  score 
of  70. 
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•  All  applicants,  either  competitive  or  non-competitive,  who  score 
70  or  greater  should  be  considered  as  eligible  for  appointment 
consideration  in  the  Air  Traffic  Control  occupation. 


•  The  Occupational  Knowledge  Test  (OKT)  should  be  administered  to 
all  competitive  and  non-competitive  applicants  together  with  the 
recommended  ATC  Test  Battery.  The  results  of  the  OKT  test  should 
be  used  for  granting  additional  earned  credit  in  place  of  the 
present  0PM  Rating  Guide. 


(1)  The  OKT  is  an  80-item  test  and  is  scored  for  correct 
answers  only.  Test  time  is  50  minutes.  There  are  no 
practice  items.  Two  minutes  are  allowed  for  readinc 
test  directions. 

(2)  The  total  test  administration  time,  including  OKT,  is 
2  hours  and  58  minutes. 


(3)  Additional  earned  credit  should  be  granted  only  to 
those  applicants  who  achieved  a  score  of  70  or  greater 
on  the  qualifying  ATC  Test  Battery  (MCAT  and  0PM  157). 

(4)  Additional  earned  credit  points  should  be  granted  to 
qualifying  applicants  based  on  OKT  scores  as  follows: 


OKT  Scores 


Right 

Transmuted 

OKT  Scores 

Scale 

0-80  0-100 


52-55 

= 

65-69 

56-59 

= 

70-74 

60-63 

= 

7  5-79 

64+ 

= 

80+ 

Additional 
Points  Earned 


3 

5 

10 

15 


(5)  For  competitive  applicants,  additional  points  based  on  OKT 
scores  should  be  added  to  the  scores  of  applicants  who 
achieve  70  or  more  on  the  ATC  Test  Battery.  The  total  score, 
together  with  any  veterans  preference  points,  should  be  used 
to  rank  candidates  for  selection  consideration  on  the  0PM 
register. 

(6)  For  non-competitive  applicants,  the  OKT  scores  together  with 
the  ATC  Test  Battery  should  be  used  as  a  basis  for  offering 
GS-7  appointments  in  the  ATC  occupation  rather  than  GS-5 
predevelopmental  appointments  to  those  individuals  whose  high 
scores  indicate  they  have  the  aptitude  and  knowledge  to 
enter  the  GS-7  Initial  ATC  Qualification  training. 


•  Qualified  applicants  should  be  ranked  for  competitive  appointment 
consideration  based  on  the  sum  of  their  scores  on  the  ATC  Test 
Battery,  the  OKT  and  Veterans  Preference  credit  with  those  appli¬ 
cants  having  the  highest  total  score  (Earned  Rating)  given  first 
consideration  for  selection. 
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The  existing  OPM  register  of  qualified  applicants  (approximately 
4,500)  should  be  retained.  Applicants  who  qualify  on  the  basis 
of  the  new  ATC  Test  Battery  should  be  interspersed  in  their 
appropriate  rank  order  with  those  applicants  who  are  presently 
on  the  register. 

The  six  parallel  forms  of  MCAT  and  the  eight  parallel  forms  of 
OKT  which  have  been  developed  and  used  in  conjunction  with  the 
validation  studies  should  be  placed  into  operational  use  for  ATC 
applicant  screening  and  selection. 

The  FAA,  Office  of  Aviation  Medicine,  should  be  delegated  the 
responsibility  from  OPM  for  continued  development  of  additional 
parallel  forms  of  MCAT  and  OKT  to  insure  against  compromise  of 
operational  test  forms. 
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